operational monitoring and automation for zvm zlinux 20150303 › share › 124 ›...

IBM z Systems Software

© 2013, 2015 IBM Corporation

Tracy Dean, [email protected]

March 2015

Operational Monitoring and Automation of z/VM and Linux on z SystemsUsing IBM Operations Manager for z/VM

SHARE, Session #16467

1

IBM Software

© 2013, 2014 IBM CorporationOperational Monitoring and Automation Scenarios for a z/VM Cluster and Linux on z Systems GuestsOperational Monitoring and Automation of z/VM and L inux on z Systems2 Operational Monitoring and Automation of z/VM and L inux on z Systems © 2013, 2015 IBM Corporation

Notices and DisclaimersCopyright © 2015 by International Business Machines Corporation (IBM). No part of this document may b e reproduced or transmitted in any form without written permissi on from IBM.

U.S. Government Users Restricted Rights - Use, dupli cation or disclosure restricted by GSA ADP Schedule Contract with IBM.

Information in these presentations (including infor mation relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of in itial publication and could include unintentional t echnical or typographical errors. IBM shall have no responsibil ity to update this information. THIS DOCUMENT IS DI STRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED . IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION, IN CLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OP PORTUNITY. IBM products and services are warranted according to the terms and conditions of the agreem ents under which they are provided.

Any statements regarding IBM's future direction, in tent or product plans are subject to change or with drawal without notice.

Performance data contained herein was generally obt ained in a controlled, isolated environments. Cust omer examples are presented as illustrations of how those custome rs have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.

References in this document to IBM products, progra ms, or services does not imply that IBM intends to make such products, programs or services available in all cou ntries in which IBM operates or does business.

Workshops, sessions and associated materials may ha ve been prepared by independent session speakers, a nd do not necessarily reflect the views of IBM. All material s and discussions are provided for informational pu rposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual particip ant or their specific situation.

It is the customer’s responsibility to insure its own compliance with legal requirements and to obtai n advice of competent legal counsel as to the identification an d interpretation of any relevant laws and regulator y requirements that may affect the customer’s business and any actions the customer may need to take to comply with such l aws. IBM does not provide legal advice or represent or warrant th at its services or products will ensure that the cu stomer is in compliance with any law.

IBM Software


Notices and Disclaimers (con’t)Information concerning non-IBM products was obtaine d from the suppliers of those products, their publis hed announcements or other publicly available sources. IBM has not tested those products in connection wi th this publication and cannot confirm the accuracy of performance, compatibility or any other claims rel ated to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppli ers of those products. IBM does not warrant the quality of any third-party products, or the ability of any su ch third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOS E.

The provision of the information contained herein i s not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or ot her intellectual property right.

� IBM, the IBM logo, ibm.com, Bluemix, Blueworks Live , CICS, Clearcase, DOORS®, Enterprise Document Management System™, Global Business Services ®, Glob al Technology Services ®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Ne tcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureC overage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QR adar®, Rational®, Rhapsody®, SoDA, SPSS, StoredIQ, Tivoli®, Trusteer®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Mach ines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current li st of IBM trademarks is available on the Web at "Copyr ight and trademark information" at: www.ibm.com/legal/copytrade.shtml .

IBM Software


Agenda

� Where do the IBM products fit

� What other sessions are available at this conference

� Introduction to recommended practices and examples

� Overview of product being used– IBM Operations Manager for z/VM

� Considerations for z/VM Single System Image– For your reference

� Recommended practices in detail– Live demonstrations

• Configuration and sample code

� Summary– Reference information– Additional demos

• Configuration options and sample code for all demos

� Hands-on lab for all products in new IBM Infrastruc ture Suite for z/VM and Linux– Wednesday, 4:30pm-5:30pm– Location: Redwood (Level 2)

IBM Software


Application Owner

Middleware Administrator

Linux System Administrator

z/VM System Programmer

z/VM SystemLinux Virtual

MachineMiddleware Applications

Three Dimensions of Systems Management

5

Who

is d

oing

the

man

agin

g?

What are they managing?

IBM Software


Application Owner

Middleware Administrator

Linux System Administrator

z/VM System Programmer

z/VM SystemLinux Virtual

MachineMiddleware Applications

Three Dimensions of Systems ManagementW

ho is

doi

ng th

e m

anag

ing?

What are they managing?

��

��

� � �

�

�

IBM Software


Managing z/VM and Linux on z Systems

� Security– RACF and zSecure Manager for z/VM

� Performance monitoring– OMEGAMON XE on z/VM and Linux

� Operational monitoring and automation– Operations Manager for z/VM– Including integration with existing monitoring and alert systems

� Backup and recovery– Backup and Restore Manager for z/VM– Tape Manager for z/VM– Tivoli Storage Manager

� Interactive provisioning and system resource manage ment– IBM Wave for z/VM

7

IBM Software


IBM Infrastructure Suite for z/VM and Linux

� New IBM bundle/suite

� Announced and available September 2014

� Tools needed to manage the z/VM and Linux on z Syst ems infrastructure– Wave for z/VM – OMEGAMON XE on z/VM and Linux– Operations Manager for z/VM– Backup and Restore Manager for z/VM

• Order Tape Manager for z/VM separately if plan to back up to tape– Tivoli Storage Manager Extended Edition

� Discounted price as a bundle

� Website:– http://www.ibm.com/software/products/en/ibm-infrastructure-suite-for-zvm-and-linux

� DeveloperWorks Wiki– https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W9b51

1b099ded_4e32_abfb_ed8ce4da5b17

8

IBM Software


IBM Wave for z/VM and Operations Manager for z/VM� IBM Wave for z/VM provides an interactive GUI interf ace for:

– Provisioning of Linux guests– Basic performance information– Monitoring of virtual server resources– z/VM and Linux administrator tasks

� Operations Manager for z/VM provides operational mo nitoring & automation– In the background

• Monitoring of console messages for z/VM service machines and Linux guests• Monitoring “state” information for z/VM service machines and Linux guests• Monitoring spool and page space on the z/VM system• Automated responses to these monitors when they are triggered

– Email– SNMP alerts– Integration with IBM Tivoli Netcool/OMNIbus enterprise alert system– Actions that address the problem immediately in addition to or instead of alert notification

– Interactive when needed• View and interact with live service machine and Linux guest consoles• View and manage spool files

� Complementary solutions– Use Operations Manager to monitor Wave service machines– Use Operations Manager to automatically initiate tasks in Wave via the Wave CLI

9

IBM Software


IBM Wave for z/VM and OMEGAMON XE on z/VM and Linux� IBM Wave for z/VM provides point in time monitoring of virtual server resources from

a single graphical interface

� OMEGAMON XE on z/VM and Linux provides – Deeper level monitoring of z/VM

– Deeper level monitoring of individual Linux guest environments– Ability to set service level thresholds and generate events when exceeded– Historical view of monitoring data

� Both OMEGAMON XE on z/VM and Linux and IBM Wave can coexist in customer environments

� Both gather the data from the Performance Toolkit for z/VM

10

IBM Software


Operational Monitoring and Automation

Reactive Proactive Predictive

IBM Software


Recommended Practices – Operational Monitoring and Automation

View and issue commands on consoles of Linux guests and CMS service machines

� Operations staff monitoring multiple consoles or a central console of alerts

� System programmers debugging a problem on a guest or service machine

12

Generate alerts and/or automatically recover from

� Abend, termination, or error messages

� Service machine disks approaching full

� Critical user IDs or guests being logged off or entering error state

� Spool and/or page space approaching full

Schedule automated system maintenance procedures

� Spool cleanup based on policies

� Minidisk cleanup (from logs), including archiving

� Orderly startup and shutdown

� Relocation of critical guests to another SSI member

� Backups of z/VM system

Keep monitoring close to the

operating system

IBM Software


Recommended Practices – Operational Monitoring and Automation




13












Reactive

Proactive

Predictive ??

IBM Software


Automation Demos Available1. View consoles of Linux guests, Linux syslog data, a nd CMS user IDs or service machines

2. Send an e-mail based on a console message

3. Send an alert to Netcool/OMNIbus based on a console message, hold and unhold messagesa. Using POSTZMSG interface to Netcool/OMNIbus b. Using SNMP interface to Netcool/OMNIbus

4. Send a message or email if spool approaches fulla. Send a message if spool usage is too high on any member of an SSI Cluster b. Send an email if spool usage is too high on a single system

5. View and clean up spool files

6. Automated spool cleanup

7. Archiving DIRMAINT’s log files when disk gets full

8. Process a file of test messages as a console

9. Process Linux syslog data as a console

10.Create a central operations console on one z/VM syst em

11.Create a central operations console across multiple z/VM systemsa. When the systems are in an SSI cluster b. When the systems are not in an SSI cluster

12. Integration with OMEGAMON XE on z/VM and Linux - take action based on CPU usage of Linux guest

13.Monitor service machines for logoff – and autolog th em

14.Send an email if page space approaches full

15.Monitor SSI connectivity between 2 cluster members

16.Suppress passwords on Linux consoles

17.Autolog a Linux Guest and Send Message if Doesn’t S tart Successfully

14



Product OverviewIBM Operations Manager for z/VM

IBM Software


Operations Manager for z/VM

16

Operations Manager for z/VM

Increase productivity� Authorized users to view and interact with monitored

virtual machines without logging onto them� Multiple users view/interact with a virtual machine

simultaneously

Improve system availability� Monitor virtual machines and processes� Take automated actions based on console messages� Reduce problems due to operator error

Service Virtual Machine being

monitored

Service Virtual Machine being

monitored

Console monitoring

Console monitoring

Take action

• View & interact with consoles

• View spool files

Automation� Routine activities done more effectively with

minimal operations staff� Schedule tasks to occur on a regular basis

Schedule tasks

Monitor page and spool usage

Respond to system events(user state changes)

Integration� Fulfill take action requests from performance monitoring

products (e.g. OMEGAMON XE on z/VM and Linux)� Send alerts to email, central event management systems

(e.g. Netcool/OMNIbus), etc.

Idle monitor

IBM Software


Features and Functions

� Monitor service machine consoles

� Monitor page space and spool usage

� Monitor system events

� Schedule events/actions

� Take actions automatically based on monitoring resu lts

� View and interact with monitored consoles from auth orized user IDs

� Find and view spool files

� Dynamic configuration

� Separation of access control

17

IBM Software


Dynamic Configuration

� Initial configuration file loaded at startup

– May imbed other configuration files– Filename can be a substitution variable for the system name

� Most configuration options can be updated while Ope rations Manager is running

– Add, delete, or change:• Rules, actions, monitors, schedules, holidays, groups, user authorization

– Suspend or resume rules, monitors, schedules

� Multiple methods

– CMS command interface– (Re)load a new or updated configuration file– Commands in action routines

� Sample configuration files provided– Includes some of the demos in this presentation

• Operations Manager configuration statements• Sample REXX code

18



View and Issue Commands on ConsolesLinux Guests and CMS Service Machines

Generate Alerts and/or Automatically Recover FromAbend MessagesTermination MessagesError Messages

IBM Software


Monitor Service Machine Consoles

20

UnfilteredFiltered

Operations Manager

OPERATOR

LINUX

TCPIP

Daily logOPER Message 1LNX Message 1LNX Message 2TCP Message 1slog Message 1OPM Message 1

Data space 6

OPER Message 1LNX Message 1LNX Message 2TCP Message 1slog Message 1OPM Message 1TEST Message 1OPER Message 2…

Test Data

syslog data

OPER Message 1OPER Message 2OPER Message 3…

TCP Message 1TCP Message 2TCP Message 3…

LNX Message 1LNX Message 2LNX Message 3…

TEST Message 1TEST Message 2TEST Message 3…

slog Message 1slog Message 2slog Message 3…

Data space 2

OPER Message 1OPER Message 2…

Data space 1

TEST Message 1TEST Message 2…

Data space 3

LNX Message 1LNX Message 3…

Data space 5

slog Message 1slog Message 2slog Message 3…

Data space 4

TCP Message 1TCP Message 2…

IBM Software


View and Interact with Consoles� Authorized users can view live consoles of monitore d service machines & guests

– Multiple users can view the same console simultaneously– No need to logon to the user ID to see its console

• No interruption of the user ID– No need to create and close console files of disjointed data– Test data and Linux syslog data treated as a “console”– Views can be defined to look at a group of consoles in one view– Can specify a date and time range for your view within currently available data– Can request a copy of the current console data for a user or set of users– Format of date in the view is based on requestor’s CP DATEFORMAT setting

� Full screen mode– Scroll up and down to view and search historical data– Auto scroll (on or off) as new output is displayed on the console– From command line, issue commands back to the monitored console

� Amount of data that is visible depends on specified or default data space size– Or date/time range specified

� Rules/actions may modify the view– Suppress messages from the console – Hold or highlight messages with color, blinking, etc.

� Authorized users can view the log file – Can also request a copy of the log file from today or a previous day

21

IBM Software


Monitor Service Machines

� Define rules to

– Scan console messages for text matching

• Includes column, wildcard, and exclusion support• Optionally restrict to specific user ID(s)

– Take actions based on matches

� Multiple rules can apply to one message

– Rules processed in order of definition in the configuration file

– FINAL option available to indicate no additional rules should be evaluated

22

IBM Software


Executing Actions

� Define action(s) to be triggered

– Specify action to take as part of the console rule definition• Action is taken when match is found

– Types of actions• Change color, highlight, hold, or suppress a console message• CP or CMS commands• REXX EXECs• Write data out on a TCP/IP port

– E.g. send data to a syslog daemon/server

� Dynamically include data about the triggering event in the action

– Available to the action via substitution variables

� Take multiple actions based on one message

– Chain actions together

– Limit the number of times an action is taken in a specified period of time

23



Generate Alerts and/or Automatically Recover FromCritical User IDs or Guests Logging OffCritical User IDs or Guests Enter Error State

IBM Software


Respond to System Events

� Create monitors for z/VM system events (*VMEVENT) r elated to user IDs– Class 0

• 0 - Logon• 1 - Logoff• 2 - Failure condition (including CP READ and Disabled Wait)• 3 - Logoff timeout started• 4 - Forced sleep started• 5 - Runnable state entered (VM READ)• 6 - Free storage limit exceeded• 9 - Outbound relocation started• 10 - Inbound relocation started• 11 - Outbound relocation complete• 12 - Inbound relocation complete• 13 - Outbound relocation terminated• 14 - Inbound relocation terminated• 15 – Timebomb exploded

� Additional classes also supported

� Optionally restrict to specific user ID(s)

� Specify the action associated with the event– Actions specified are the same as those for schedules, console rules, and other monitors

25



Generate Alerts and/or Automatically Recover FromSpool Space Approaching FullPage Space Approaching Full

IBM Software


Monitor Page and Spool Usage, View Spool Files

� Create page and spool space monitors to trigger act ions when– Percent of spool usage falls within a specified range– Percent of spool usage increases at a specified rate– Percent of page space usage falls within a specified range– Percent of page space usage increases at a specified rate

� Actions triggered can be the same actions used by c onsole monitoring

� For spool files, authorized users can – Display a list of spool files based on one or more attributes

• Owner• Size• Date created

– From the list, the user can• Sort the list on any of the available columns• View the contents of an individual spool file• Purge, transfer, or change a spool file

27



Schedule Automated System Maintenance Procedures

Monitor for Rules, Monitors and Schedules that Were Not Triggered

Spool Cleanup Based on PoliciesBackupsDisk CleanupOrderly Startup and Shutdown

IBM Software


Schedule Events and Actions

� Define schedules– Hourly, daily, weekly, monthly, or yearly, nth weekday of the month– Once on specified month, day, year, and time– Based on ISO week definitions (week number; even, odd, first, last week) – At regular intervals

• Every x hours and y minutes– Within a specified window of time

• Specify start time• Specify conflicting schedules• Specify maximum time to defer this schedule

– Within limits• Restrict to specific days of the week: Monday through Sunday plus holidays• Restrict to certain hours of the day

� Specify the action associated with the schedule– Actions specified are the same as those for console rules and all other monitors

29

IBM Software


Idle Monitors

� Define idle monitors

– Watch for idle rules, schedules, and monitors

• Rule, schedule, or monitor not triggered n number of times within specified period of time

� Specify the action associated with the idle monitor

– Actions specified are the same as those for schedules, console rules, other monitors

30



SSI vs non-SSI Considerations

IBM Software


Operations Manager - non-SSI Environment

32

z/VM

Main Server (OPMGRM1)

Existing z/VM User

Existing Linux Guest

Existing Linux Guest

� Captures consoles

� Evaluates rules

� Triggers schedules

� Monitors events, page space, and spool usage

� Executes actions or sends them to action processing servers

Authorized Users

� View and interact with monitored consoles

� Find and view spool files

� Update configuration information

Action Processing Server (OPMGRSn)

� 0 to n server instances

� Processes actions as a result of:� Console rule matching� Spool monitors� Event monitors� Schedules

Remote system (z/VM, email,

Netcool\OMNIbus, etc.)



SSI ConsiderationsConsole Monitoring

IBM Software


SSI Considerations for Console Monitoring

34

ZVMA


TCPIP LINUX2

LINUX1OPERATOR

ZVMA


TCPIP LINUX2

LINUX1OPERATOR

ZVMB

TCPIP LINUX4

LINUX3OPERATOR

ZVMB


TCPIP LINUX4

LINUX3OPERATOR

ConfigToolsExecs

ConfigToolsExecs

Option 1

Option 2Recommended

ADMIN1

ADMIN1

Single Config User

Multiconfig User

IBM Software


Operations Manager in SSI Cluster - Example

35

z/VM - MEMBERA


OPERATOR

RHEL5G

DEMOADMN

OPMGRSn

z/VM - MEMBERB


OPERATOR

SLES11C

� Multiconfiguration users: OPMGRM1, OPMGRSn, OPERATOR, MAINT

� Single configuration users: RHEL5G, SLES11C, OPERSSI, DEMOADMN� May relocate OPERSSI and DEMOADMN manually (supported) or via VMRELOCATE

(unsupported, but you can make it work)

OPMGRSn

OPERSSI(central console) OPERSSI

(central console)

RHEL5G

Remote system (z/VM, email,

Netcool\OMNIbus, etc.)

DEMOADMN

IBM Software


Relocating OPERSSI and DEMOADMN (CMS Users) …

� VMRELOCATE for CMS user IDs not officially supporte d

� Can be done for some CMS users– Create single configuration user ID for z/VM system disks– Copy MAINT 190, 19D, 19E to minidisks owned by this new user ID

• Use DDR for 190 since it needs to be IPLable– Relocateable CMS user must IPL from identical NSS (CMS) or minidisk (190)

• Use SPXTAPE to copy CMS NSS– VMRELOCATE uses checksum of NSS to determine if identical– CMS NSS includes date/time it was loaded

• Or, have relocateable CMS users IPL 190 instead of IPL CMS

36

OPERSSI DIRECTUSER OPERSSI ……OPTION CHPIDVIRTUALIZATION ONE…IPL 190…LINK CMAINT 0190 0190 RRLINK CMAINT 019D 019D RRLINK CMAINT 019E 019E RR…

PROFILE EXEC

/* PROFILE EXEC for OPERSSI */…‘SET RELPAGE OFF’…

IBM Software


… Relocating OPERSSI and DEMOADMN (CMS Users)

� Beware– It’s worth repeating … VMRELOCATE for CMS user IDs

not officially supported

– All members of the cluster must be kept at same z/VM (or at least CMS) code level

– If IPL 190, will use more memory as each user ID will have private copy of CMS

– SET RELPAGE OFF may have a negative impact on overall system performance

– Only works for “basic” CMS users• All relocation rules still apply• E.g. user IDs connecting to VMCF or IUCV can’t relocate

37

IBM Software


Monitor Service Machines - Considerations� Consoles received by Operations Manager via SECUSER or OBSERVER

– Prefer SECUSER• OBSERVER won’t detect CP and VM READ messages• Output of actions on OBSERVEd console may not be viewable in console

– OBSERVER allows Operations Manager to receive console output even if user is logged on

� Single System Image allows SECUSER and OBSERVER acr oss members of cluster– Content does not contain member name information– Rules, actions, and users wouldn’t be able to distinguish between IDENTITY users on multiple

members– Creates single point of failure on one member

� Recommendation for z/VM V6.2 or V6.3 Single System Image environments– Have all consoles monitored by an Operations Manager server on the same member as the

monitored guest (i.e. all Operations Manager servers are IDENTITY users)• Requires action processing servers (OPMGRSn) to be on same member as main server

– Share configuration data on minidisk owned by single configuration user • For example: VMTOOLS 198• Master configuration file unique to each member• Imbed common file(s) used by all members

– Request a copy of the current console of a remote user• SMSG OPMGRM1 at membername VIEWCON USER(userid),MOD E(RDR)

38



SSI ConsiderationsPage Space MonitoringSpool Space MonitoringViewing and Managing Spool Files

IBM Software


SSI Considerations for Spool and Page Space Monitoring

40

ZVMA


LINUX1OPERATOR

ZVMB


LINUX3 OPERATOR

ADMIN1

Single Config User

Multiconfig User

Page Space ZVMA

Page Space ZVMB

TCPIP LINUX2 TCPIPLINUX4

OPERATORA

TCPIPA

ADMIN1A

LINUX1A

Spool Volume ZVMA

LINUX4A

OPERATORB

TCPIPB

LINUX3B

Spool Volume ZVMB

LINUX2A

LINUX3A

ADMIN1B

LINUX1B

LINUX4B

LINUX2B

Percent full

LINUX1

ADMIN1

LINUX2

LINUX1

ADMIN1

LINUX3

LINUX4LINUX2

LINUX1

ADMIN1

IBM Software


Spool and Page Space Monitoring - Considerations

� Page space is local

– Separate space for each member and only visible to the local member

– No impact from SSI

� Spool data

– Spool files are placed on spool volumes owned by the member where the spool file was created

– Users see their own spool data no matter where they are logged on and where the data was created

41

IBM Software

© 2013, 2014 IBM CorporationOperational Monitoring and Automation Scenarios for a z/VM Cluster and Linux on z Systems Guests42 Operational Monitoring and Automation of z/VM and L inux on z Systems © 2013, 2015 IBM Corporation


� Users and applications (like Operations Manager) wh o can see all spool files need to be aware:– Spool data for multiconfiguration users

• Only spool files owned by the local instance of that user are visible on the local member

• No visibility to spool files owned by other instances of that user on other members

– Spool data for single configuration users:

Single configuration user Status

Spool files created on this member

Spool files created on other members

User logged off Visible Not visible

User logged onto this member

Visible Visible (but not on local spool volumes)

User logged onto another member

Visible Not visible

42

IBM Software



� Recommendation

– Have an Operations Manager server on each member to monitor spool and page space

– Be aware of spool files visible in Operations Manager but not resident on this member’s spool volumes

• Indicated with ”+” in VIEWSPL

43



SSI ConsiderationsManaging Configuration Files

IBM Software


Managing Configuration Files� Put all configuration files on a shared disk

– Minidisk owned by a single configuration user (not an Operations Manager service machine)– SFS

� Create a common configuration file used by all memb ers– All Operations Manager servers on all members load this file

� Imbed a unique configuration file based on the syst em name of this member

� Request configuration file reload from user IDs on other members of a cluster• Use SMSG OPMGR1 at <member> CONFIG …

45

* Configuration specific to MEMBERA system

…

MEMBERA CONFIG E* Configuration specific to MEMBERB system

…

MEMBERB CONFIG E

* Common configuration for all member of the cluste r

….

CONFIG FN(&SYSNAME),FT(CONFIG),FM(E)

OPMGRM1 CONFIG E



SummaryReferencesDemos – Including Screenshots, Configuration Statements, and REXX

IBM Software


Recommended Practices – Operational Management

47















RulesSchedules

Spool/Page monitors

Schedules

Rules, Archive Mgr

SFPURGER

Rules, monitors

Backup Manager

IBM Software


Summary

� Use Operations Manager to – Automate daily operations– Integrate your z/VM and Linux on z Systems environment with existing

enterprise monitoring and alerting– Prevent problems rather than react to them– Automate reactions to problems when they can’t be prevented– Improve problem determination procedures– Increase programmer and operator productivity– Continue to monitor locally with improved management of clusters

� Sometimes several alternatives for monitoring for t he same event– Console message (rules)– Scheduled healthchecks (schedules)– User ID status changes (event monitor)

� Actions allow integration with other platforms and products

48

IBM Software


Reference Information

� Web sites– Product page: http://www.ibm.com/software/products/en/operations-manager-for-zvm

• Publications, presentation, white papers• Pre-requisites• Support

– Infrastructure Suite wiki: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W9b511b099ded_4e32_abfb_ed8ce4da5b17• Videos• One page flyer• Product information

� e-mail– Mike Sine, [email protected], Advanced Technical Skills (ATS)– Tracy Dean, [email protected], Product Manager

� White papers on Operations Manager website (Resourc es tab)– Routing Linux syslog data– Sending alerts from Operations Manager to Netcool/OMNIbus– Using Shared File System to store Operations Manager configuration files and automation EXECs– Automatically logging on a user at Linux system boot time for easier console management and action

execution– Enabling the FACILITY Class for Use by RACF for z/VM



Demonstration Scenarios

IBM Software




3. Send an alert to Netcool/OMNIbus based on a console message, hold and unhold messagesa. Using POSTZMSG interface to Netcool/OMNIbusb. Using SNMP interface to Netcool/OMNIbus

4. Send a message or email if spool approaches fulla. Send a message if spool usage is too high on any member of an SSI Cluster – see how spool files appear in SSIb. Send an email if spool usage is too high on a single system









13.Monitor service machines for logoff – and autolog the m




17.Autolog a Linux Guest and Send Message if Doesn’t St art Successfully

51

IBM Software

© 2013, 2014 IBM CorporationOperational Monitoring and Automation Scenarios for a z/VM Cluster and Linux on z Systems GuestsOperational Monitoring and Automation of z/VM and L inux on z Systems52 Operational Monitoring and Automation of z/VM and L inux on z Systems © 2013, 2015 IBM Corporation52

Thank

You

MerciGrazie

Gracias

Obrigado

Danke

Japanese

English

French

Russian

German

Italian

Spanish

Brazilian Portuguese

Arabic

Traditional Chinese

Simplified Chinese

Tamil

Thai

KoreanHindi



Demonstration Scenarios

IBM Software




3. Send an alert to Netcool/OMNIbus based on a console message, hold and unhold messagesa. Using POSTZMSG interface to Netcool/OMNIbus b. Using SNMP interface to Netcool/OMNIbus

4. Send a message or email if spool approaches fulla. Send a message if spool usage is too high on any member of an SSI Cluster – see how spool files appear in SSIb. Send an email if spool usage is too high on a single system









13.Monitor service machines for logoff – and autolog th em




17.Autolog a Linux Guest and Send Message if Doesn’t S tart Successfully

54

IBM Software


Scenario 1:View Live Consoles of Linux Guests, Linux Syslog Data, CMS Service Machines

� Configure user IDs / guests to be monitored by Oper ations Manager

� Route syslog data from a Linux guest to Operations Manager

� From authorized user, view the live console data of

– OPERATOR

• Issue VM commands

– A Linux guest

• Issue Linux commands

– Linux syslog data

55

IBM Software


Scenario 1: Detailed Steps

� From an authorized VM user ID, verify OPERATOR and Linux guest are being monitored by Operations Manager

q secuser operator

q observer sles11c

� View the console of OPERATORgomcmd opmgrm1 viewcon user(operator)

� Issue VM commands allowed by OPERATORid

cp send bkrbkup cms listfile

� View the console of the backup servergomcmd opmgrm1 viewcon user(bkrbkup)

� View the console of a Linux guest and issue Linux c ommandsgomcmd opmgrm1 viewcon user(sles11c)

echo hello world

� View the syslog data from a Linux guestgomcmd opmgrm1 viewcon user(lxsyslog)

56

IBM Software


IBM Software


Scenario 1: How Do You Do That?� For console data

– Make OPMGRM1 the secondary user of OPERATOR and SLES11D• Via CONSOLE statement in CP directory entry (recommended)• Via SET SECUSER command

� For Linux syslog data– Set up TCP/IP listener for syslog data

* DEFTCPA NAME(LNXSYSLG),+

TCPUSER(TCPIP),+ TCPAPPL(GOMRSYL),+ TCPADDR(000.000.000.000),+ TCPPORT(00514),+ PARM(LXSYSLOG03330417UTF8)

– Update TCP/IP configuration to allow Operations Manager to listen for UDP traffic on the specified port(s)

• Port 514 used here– Update the Linux guest to send its syslog data to the IP address and port of

your z/VM system – Refer to white paper on Operations Manager web site for details

64

IBM Software


Scenario 2:Send an Email if Abend or Fatal Message Occurs

� Watch all monitored consoles for an error message that includes the word “fatal” or “abend”– Message must also contain the word “mail” (for demo

purposes only)

� Send an email if one of the words appears on a console

� Dynamically include in the email– Host name of z/VM system where the error occurred

– User ID that received the error message

– Indicator of whether the word was fatal or abend

– Full text of the error message

65

IBM Software



� From any VM user ID:tell opmgrc1 this is an abend message from SHARE. S end an e-mail, please.

� From an authorized VM user ID, view the console of OPMGRC1:

gomcmd opmgrm1 viewcon user(opmgrc1)

� Check the inbox of the appropriate person to see th e e-mail

66

IBM Software


IBM Software


Scenario 2: How Do You Do That?

Rules in Operations Manager:** Send an e-mail to someone if I see a message cont aining the word

* "fatal" on any monitored console

DEFRULE NAME(FATLMAIL),+

MATCH(*FATAL*mail*),+

ACTION(EMAIL),+

PARM(FATAL)

*

* Send an e-mail to someone if I see a message cont aining the word

* "abend" on any monitored console

DEFRULE NAME(ABNDMAIL),+

MATCH(*ABEND*mail*),+

ACTION(EMAIL),+

PARM(ABEND)

70

IBM Software



71

Action in Operations Manager:*

* Replace "tld1 at us.ibm.com" with the e-mail addr ess of the user that

* should receive the e-mail

* Leave &u, &p, and &t as-is. These represent the u ser ID that had the

* "fatal" message, the parameter passed (fatal or a bend), and the

* text of the message. These will be included in t he text of the

* e-mail.

DEFACTN NAME(EMAIL),+

COMMAND(EXEC SMTPNOTE tld1 at us.ibm.com &u &p &t), +

OUTPUT(LOG),+

ENV(LVM)

IBM Software



72

SMTPNOTE EXEC (excerpts)/* */

Parse arg mail_user 'AT' mail_node baduser errtype msgtext

if errtype = 'FATAL' then

errtext = 'Fatal error on user ID' baduser 'on z/VM system'

else

if errtype = 'ABEND' then

errtext = 'Abend on user ID' baduser 'on z/VM system'

else errtext = msgtext

/* Construct the e-mail */

line.1 = 'OPTIONS: NOACK LOG SHORT NONOTEBOOK ALL CLASS A'

line.2 = 'Date: ' Date() ',' Time()

line.3 = 'From: Operations Manager for z/VM'

line.4 = 'To: ' mail_user 'at' mail_node

line.5 = 'Subject: ' errtext

line.6 = 'The following message was received on' baduser 'running on'

line.7 = msgtext

line.8 = ' '

line.9 = 'DO NOT REPLY - This e-mail was generated by an automated service machine

line.0 = 9

'PIPE stem line. | > TEMP NOTE A'

'EXEC SENDFILE TEMP NOTE A (NOTE SMTP'

IBM Software


Scenario 3a:Send an Alert to OMNIbus – Using POSTZMSG

� Watch all monitored consoles for an error message t hat includes the word “fatal” or “abend”

– Message must also contain the word “omni” (for demo purposes only)

� Send an alert to OMNIbus if one of the words appear s on a console

– Use POSTZMSG, running on Linux guest• Do not trigger the action if the message is on this guest

� Dynamically include in the alert


– Indicator of whether the word was fatal or abend

73

IBM Software


Scenario 3a: Detailed Steps

� View “All Events” in OMNIbus

� From any VM user ID:tell opmgrc1 this user is abending at SHARE. Tell O MNIBUS.

� From an authorized VM user ID, view the console of OPMGRC1:


� From an authorized VM user ID, view the console of the Linux guest that runs POSTZMSG:

gomcmd opmgrm1 viewcon user(esmts112)

� View the OMNIbus console to see the alert

74

IBM Software


IBM Software


Scenario 3a: How Do You Do That?

Rules in Operations Manager:** Send an alert to OMNIBUS for fatal errors on cons oles

DEFRULE NAME(FATLOMNI),+

MATCH(*fatal*omni*),+

EXUSER(ESMTS112),+

ACTION(ALRTOMNI),+

PARM(FATAL)

*

* Send an alert to OMNIBUS for abends on consoles

DEFRULE NAME(ABNDOMNI),+

MATCH(*abend*omni*),+

EXUSER(ESMTS112),+

ACTION(ALRTOMNI),+

PARM(ABEND)

78

IBM Software


Scenario 3a: How Did You Do That?

79


* Call POSTZMSG on a Linux guest to send alert to O MNIBUS

DEFACTN NAME(ALRTOMNI),+

COMMAND(EXEC POSTZMSG &u &p),+

OUTPUT(LOG),+

ENV(LVM)

IBM Software


Scenario 3a: How Did You Do That?

80

POSTZMSG EXEC (excerpts)/* */

Address Command

Parse arg baduser errtype

if errtype = 'ABEND' then

do

zerrtype = 'CRITICAL'

cmdpart2 = '-m guest_is_abending hostname='baduser

cmdpart4 = 'sub_origin=tcp SCARY_EVENT OpsMgr'

end

else

do

zerrtype = 'WARNING'

cmdpart2 = '-m fatal_error_on_guest hostname='badus er

cmdpart4 = 'sub_origin=tcp WARN_EVENT OpsMgr'

end

cmdpart1 = './postzmsg -f e2o.conf -r' zerrtype

cmdpart3 = 'sub_source=postzmsg origin='baduser

'CP SEND ESMTS112 cd /workloads'

'CP SEND ESMTS112' cmdpart1 cmdpart2 cmdpart3 cmdpa rt4

IBM Software


Scenario 3b:Send an Alert to OMNIbus – Using SNMP

� Watch all monitored consoles for an error message t hat includes the word “abend”

– Message must also contain the word “snmp” (for demo purposes only)

� If this word appears on a console

– Change the message to red and hold it

– Send an alert to OMNIbus, using SNMPTRAP command on z/VM

– Automatically unhold the message after 4 minutes

� Dynamically include in the alert

– IP address of the z/VM system where the error occurred


– Text of the abend message

81

IBM Software


Scenario 3b: Detailed Steps

� View “All Events” in OMNIbus

� From any VM user ID:tell esmts105 this user is abending during demo. Se nd SNMP alert to Netcool

� From an authorized VM user ID, view the console of ESMTS105 (a Linux guest):


� Issue some Linux commands so the held message moves to the top of the screen

� View the OMNIbus console to see the alert

� After 4 minutes, view the console of ESMTS105 again and notice the held message has moved off the screen


82

IBM Software


IBM Software


Scenario 3b: How Do You Do That?

Rule and actions in Operations Manager:*

* Send an alert to OMNIbus using SNMP for abend msg s on consoles

DEFRULE NAME(ABNDSNMP),+

MATCH(*abend*snmp*),+

ACTION(SNMPALRT)

*

DEFACTN NAME(SNMPALRT),+

COMMAND(EXEC SNMP2OMN &T),+

INPUT(CRE,HLD),+

ENV(SVM),+

NEXTACTN(UNHOLD),+

NEXTDELY(03:30)

*

DEFACTN NAME(UNHOLD),+

COMMAND('ALTRCON USER(ESMTS105),MATCH(*abend*snmp*) ,ELAPSED(180),HLD(N)'),+

ENV(GOM)

87

IBM Software


Scenario 3b: How Did You Do That?

88

SNMP2OMN EXEC/* SNMP2OMN action routine for Operations Mgr */

address command

parse arg ":" msgtext

msgtext2 = '"'msgtext '"'

/* Send message */

snmptrap trape 1.1 number 30 1.2 text "UXZVM001" 1.3 text msgtext2 ent 1.3.6.1.4.1.9545.6

exit

IBM Software


Scenario 3b: Additional Steps Required on z/VM

� SNMPD user ID configured and running

� Update files on TCPMAINT 198 disk

– Add OMNIbus IP address to SNMPTRAP DEST file

– Open SNMPD and SNMPQE ports in PROFILE TCPIP

– Update SNMPMIBX TEXT section of MIB_EXIT DATA

� Give OPMGRM1 and OPMGRSn access to SNMPTRAP command

– On TCPMAINT 592 disk

89

IBM Software


Scenario 3b: Additional Steps Required on OMNIbus

� Install the IBM Tivoli Netcool/OMNIbus SNMP Probe – Install it on same platform as target OMNIbus server

� Customize operational information in the probe prop erties (mttrapd.props)– Listening port, heartbeat interval, mibs and mibs locations, etc.

� Customize the probe rules (mttrapd.rules) – Map variables created by the probe (from data extracted from

the SNMP trap) into the desired OMNIbus event fields• Default mappings for the SNMP generic traps (trap types 0-5)• Enterprise-specific traps (trap type 6) require customization

� Documentation for installation and customization – IBM Tivoli Netcool/OMNIbus SNMP Probe Reference Guide

(SC23-6003-04)

90

IBM Software


Scenarios 3a and 3b – POSTZMSG vs SNMP

� Using POSTZMSG

– Can direct the alert to only the IP address(es) you specify

– Need a Linux guest running and logged on that can run POSTZMSG and must be on the same z/VM system• Can be overcome by using a socket interface to send POSTZMSG

command to the guest

– Limit of 160 characters on POSTZMSG command sent to Linux guest (using CP SEND)• Can’t always send full text of message• Can be overcome by using a socket interface to send POSTZMSG

command to the guest

� Using SNMP

– No requirement for a Linux guest. SNMP runs on z/VM.

– No limit on message size

– All SNMP alerts on z/VM go the same set of IP addresses

91

IBM Software


Scenario 4a:Send a Message if Spool Usage is Too High on Any Member in an SSI Cluster

� Operations Manager monitors the spool usage (percen t full) on each member of a cluster

– For demo purposes, spool monitor is currently suspended

– We’ll dynamically resume (re-activate) the spool monitor• Must reactivate on each member of a cluster

– Demo monitor requires spool to only be 5% full

� Usage exceeds the specified limit

� Automatically send a message to a central console f or the entire cluster

– Send a maximum of 3 messages per hour

� Message includes the member name and % full

� For demo purposes, suspend (de-activate) the spool monitors when complete

� Demonstrate which spool files are visible on each m ember

92

IBM Software



� From an authorized VM user ID, see the spool usage on local member TEST7SSI:

gomcmd opmgrm1 viewspl

� From a user ID with Operations Manager privileges:gomcmd opmgrm1 resume spool(splfull)

smsg opmgrm1 at testcssi resume spool(splfull)

� Check the Operations Manager log to see the spool m onitor triggered on local member:

gomcmd opmgrm1 viewlog

� View the central console for the cluster to see war ning messages from each member:

gomcmd opmgrm1 viewcon user(operssi)

� From a user ID with Operations Manager privileges:gomcmd opmgrm1 suspend spool(splfull)

smsg opmgrm1 at testcssi suspend spool(splfull)

93

IBM Software



� From member TEST7SSI, send a spool file to a single configuration and a multiconfiguration user:

sendfile test7 file a demoadmn op1

� From member TEST7SSI, send a spool file to a multiconfiguration user on another member:

sendfile testc file a op1 at testcssi

� From a user ID with Operations Manager privileges on TEST7SSI, view spool files on TEST7SSI:


� From a user ID with Operations Manager privileges on TESTCSSI, view spool files on TESTCSSI:


94

IBM Software


IBM Software



Rule and action in Operations Manager:*

* Send an alert to OPERSSI console if spool too ful l

DEFSMON NAME(SPLFULL),+

USAGE(005-100),+

INTERVAL(1),+

LIMIT(3,3600),+

ACTION(SPLPAGE)

*

DEFACTN NAME(SPLPAGE),+

COMMAND(EXEC MSG2OPER junk junk &0 &4 &3),+

ENV(LVM)

*

SUSPEND SPOOL(SPLFULL)

104

IBM Software



MSG2OPER EXECAddress Command Parse arg userid euser event sourcesys msgtext

'GOMGLBL INTO sysname NAME tcphostname'

if userid = '_GOMEMON' then do

if event = 9 then msgtext = 'Outbound relocation for' euser 'on' sourcesys 'started'

else msgtext = 'Inbound relocation for' euser 'on' sourcesys 'started'

'CP MSGNOH OPERSSI AT ALL From' sysname ':' msgtext end

else if event = 'SPOOL' then

'CP MSGNOH OPERSSI AT ALL Spool is' sourcesys'% full on' sysname else if event = 'PAGE' then

'CP MSGNOH OPERSSI AT ALL Page space is' sourcesys'% full on' sysname else

'CP MSGNOH OPERSSI AT ALL From' userid 'on' sysname ':' msgtext

Exit rc

105

IBM Software


Scenario 4b:Send an Email if Spool Usage is Too High

� Operations Manager monitors the spool usage (percen t full)

– For demo purposes, spool monitor is currently suspended

– We’ll dynamically resume (re-activate) the spool monitor

– Demo monitor requires spool to only be 5% full or higher


� Automatically send an e-mail to someone who can eva luate and take action

� For demo purposes, suspend (de-activate) the spool monitor when complete

106

IBM Software



� From an authorized VM user ID, see the spool usage:gomcmd opmgrm1 viewspl

� From a user ID with Operations Manager privileges:gomcmd opmgrm1 resume spool(splfull)

� Check the Operations Manager log to see the spool monitor triggered:


� Check the inbox of the appropriate person to see th e e-mail

� From a user ID with Operations Manager privileges:gomcmd opmgrm1 suspend spool(splfull)

107

IBM Software


IBM Software



Spool monitor and action in Operations Manager:**

DEFSMON NAME(SPLFULL),+

USAGE(025-100),+

INTERVAL(1),+

LIMIT(3,3600),_

ACTION(SPLEMAIL)

*

DEFACTN NAME(SPLEMAIL),+

COMMAND(EXEC SMTPSPL tld1 at us.ibm.com &4),+

ENV(LVM)

113

IBM Software



114

SMTPSPL EXEC (excerpts)

/* */Parse arg mail_user dummyat mail_node spoolpct

errtext = 'Spool is' spoolpct'% full on z/VM system'

/* Get TCP hostname and domain from Ops Mgr global variables */ …line.1 = 'OPTIONS: NOACK LOG SHORT NONOTEBOOK ALL CLASS A' line.2 = 'Date: ' Date() ',' Time() … line.8 = errtext 'on' fqdomain_name line.9 = ' ' line.10 = 'Following are the top ten largest spool files and the top ten uith the most spool files.' line.11 = '===============================================================' line.0 = 11 'PIPE stem line. | > TEMP NOTE A' 'PIPE command GOMCMD OPMGRM1 STATUS DETAIL(SPOOLUSR) | specs words 4-* 1 |spooldata.' spooldata.0 = 20'PIPE stem spooldata. | >> TEMP NOTE A''EXEC SENDFILE TEMP NOTE A (NOTE SMTP'

IBM Software


Scenario 5:Find and View Spool Files – Clean up the Spool

� Authorized user specifies spool search criteria

– By user ID

– By date

– By file size

� Result list presented

– Sort

– Open/view a specific spool file

– Purge, modify metadata, or transfer a file

115

IBM Software



� From an authorized VM user ID, view the spool files:gomcmd opmgrm1 viewspl

� Sort by date

– Put cursor on date column header and hit F6

� Find the spool files just sent and type PURGE next to them

� From an authorized VM user ID, view the log to see that the spool monitor is no longer triggered:


116

IBM Software


IBM Software


Scenario 6:Automated Spool Clean Up

� Use z/VM SFPURGER utility to manage spool files based on criteria, e.g.

– User ID

– Days in spool

– Class

– Number of records

� Automate SFPURGER execution

– Regularly scheduled using Operations Manager

– Triggered by Operations Manager spool monitor

122

IBM Software



� From an authorized VM user ID, view the spool files for a specific user:

gomcmd opmgrm1 viewspl user(tstadmn2)

� Send a file to this user as class Zsendfile profile exec a tstadmn2 (class z

� View spool files for this user again to see the new filegomcmd opmgrm1 viewspl user(tstadmn2)

� Delete any existing schedules called DEMOgomcmd opmgrm1 delschd name(demo)

� Schedule SFPURGER for execution– It will purge any files of class Z

gomcmd opmgrm1 defschd name(demo),action(sfpurger),WHEN(now)

� View spool files for this user again to see the new file is gonegomcmd opmgrm1 viewspl user(tstadmn2)

123

IBM Software


IBM Software



Action in Operations Manager to call z/VM’s SFPURGER EXEC*DEFACTN NAME(SFPURGER),+

COMMAND(EXEC SFPURGER FORCE),+

OUTPUT(LOG),+

ENV(LVM)

SFPURGER OPTIONS file

* Send console log to user ID TSTADMN1 at demo node

CONSOLE TSTADMN1 DEM1ZVM

* Erase LOG and RUN files that are more than 3 days old

KEEPDAY 21

* Set prime shift start and end times

PRIMSHFT 07:30:00 16:30:00

* Use defaults for the following:

* MSGTYPE SORTMOD SFPCNTL SOSCNTL SFPMOD APPEND

SFPCNTL SFPTRACY

131

IBM Software



SFPTRACY CONTROL

* Ignore any spool files found in the NSS queue (pr ivilege class E)

QUEUE NSS A CTION IGNORE

*

* Purge any spool files found in class Z

CLASS Z A CTION PURGE

Make sure OPMGRM1 links and accesses MAINT 193 disk for access to SFPURGER functions

132

IBM Software


Scenario 7:Detecting Disk Full Conditions of Logging IDs

� Operations Manager monitors the console of a user I D that does logging

– DIRMAINT, for example

� Disk full or early warning message triggers a rule/ action in Operations Manager

– Quiesce or shut down DIRMAINT

– Send the log files to a separate service machine

– Erase the log files from DIRMAINT’s logging disk

– Restart DIRMAINT

– Separately, other service machine automatically archives all files it receives (in Archive Manager for z/VM)

– Log files are safely archived in Archive Manager and DIRMAINT is running with a clean log disk

� Get a copy of the console for further review/debugg ing

133

IBM Software



� From an authorized VM user ID, view the DIRMAINT co nsole:gomcmd opmgrm1 viewcon user(dirmaint)

� In the console view– Issue CMS commands to copy old (large) log files to DIRMAINT’s log disk

cms copyfile dirmaint tlog0914 t = tlog0912 h

– Verify the logging disk is more than 75% fullcms q disk

– Run DIRMAINT’s hourly processing nowexec dvhourly

– Verify the logging disk is less than 75% fullcms q disk

� Exit the console view and find the files in the arc hiveamvlist

– Type “archlogs” in the owner field and press ENTER

� Request a copy of the console for further review/de bugginggomcmd opmgrm1 viewcon user(dirmaint),mode(rdr)

134

IBM Software


IBM Software



Console rule and action in Operations Manager:DEFRULE NAME(DIRMLOG),+

MATCH(*DVHRLY3895*01AA*),+

USER(DIRMAINT),+

ACTION(DIRMLOG)

*

DEFACTN NAME(DIRMLOG),+

INPUT(AHI),+

NEXTACTN(DIRMLOGB)

*

DEFACTN NAME(DIRMLOGB),+

COMMAND(EXEC DIRM1AA &U),+

ENV(LVM)

Authorize Operations Manager to issue DIRM SHUTDOWN – from MAINT issueDIRM AUTHFOR OPMGRM1 CMDLEVEL 150A CMDSET O

141

IBM Software



DIRM1AA EXEC (excerpts):Parse Upper Arg Tuser . ;

/* Try to shut DIRMAINT down. */

Say 'DIRM1AA - Issuing DIRM SHUTDOWN ....';

Address CMS 'DIRM SHUTDOWN';

Address Command 'CP LINK' Tuser '1AA' Dev 'MR';

Address CMS 'ACCESS' Dev Fm;

Address Command 'PIPE CMS LISTFILE DIRMAINT *LOG*' Fm '( NOHEADER',

'| STEM FILES.';

Do I = 1 to Files.0;

Parse Upper Var Files.I Fn Ft .;

Address CMS 'SENDFILE' Fn Ft Fm 'TO ARCHLOGS';

If Rc = 0 then Do;

Sent = Sent+1;

Address CMS 'ERASE' Fn Ft Fm;

End

Address Command 'CP XAUTOLOG' Tuser;

142

IBM Software


Scenario 8:Process a File of Test Messages as a Console

� Create a file containing lines of test messages

– Test rules and actions without creating critical conditions

� Use Operations Manager to send the file for processing

– Treat it as the console of one user

– Send it again treating it as the console of another user

– Notice triggered rules and actions are different

� View the “consoles” of these two users

143

IBM Software



� Create or view a file of test messages

xedit test consdata a

– Notice the “hello” message in the file

� From a z/VM user ID, send the test file to Operatio ns Manager

– Send it twice, specifying two different “owning” user IDs. One generates a message and one doesn’t:

gomrsif test consdata a 9.39.64.72 63000 tstadmn8

gomrsif test consdata a 9.39.64.72 63000 tstuser8

� From an authorized z/VM user ID, view the consoles of the owning user IDs:

gomcmd opmgrm1 viewcon user(tstadmn8)

gomcmd opmgrm1 viewcon user(tstuser8)

144

IBM Software


IBM Software



Console rule and action in Operations Manager:*

DEFRULE NAME(TESTEX),+

MATCH(*HELLO*),+

MCOL(001:030),+

ACTION(TESTEX),+

EXGROUP(TSTUSERS)

*

DEFACTN NAME(TESTEX),+

COMMAND(CP MSG TSTADMN1 HELLO BACK FROM &U.),+

OUTPUT(LOG),+

ENV(LVM)

149

IBM Software



Set up TCP/IP listener for test data and define gro up of consoles:*

DEFTCPA NAME(TESTDATA),+

TCPUSER(TCPIP),+

TCPAPPL(GOMRSIF),+

TCPADDR(000.000.000.000),+

TCPPORT(63000)

*

DEFGROUP NAME(TSTUSERS),+

USER(TSTUSER*)

Update TCP/IP configuration to allow Operations Man ager to listen on the specified port

150

IBM Software


Scenario 9:Process Linux Syslog Data as a Console

� Route syslog data from a Linux guest to Operations Manager for z/VM– Supports syslogd, syslog-ng, rsyslog

– syslog-ng and rsyslog include hostname or IP address in message

� Treat it as the console of a “fake” user ID

� Trigger rules and actions based on syslog data

� View the “console” containing syslog data

� Option to create one console per syslog or combine multiple syslogs into one console

151

IBM Software



� From an authorized z/VM user ID, view any syslog da ta already received

gomcmd opmgrm1 viewcon user(lxsyslog)

� Use PUTTY to connect to a Linux guest

� Login as root and issue the commandlogger here is a critical test message from SHARE

� Return to the VIEWCON session

– See the message in the syslog “console”

– Using syslog, so no hostname or IP address

� Repeat from a different Linux guest that uses syslo g-ng

152

IBM Software


IBM Software




DEFRULE NAME(LXLOG),+

MATCH(*critical test message*),+

ACTION(LXLOG),+

USER(LXSYSLOG)

*

DEFACTN NAME(LXLOG),+

COMMAND(CP MSG TSTADMN1 Got a critical message '&T' from &U.),+

OUTPUT(LOG),+

ENV(LVM)

161

IBM Software


Scenario 9: How Do You Do That?� Set up TCP/IP listener for syslog data* DEFTCPA NAME(LNXSYSLG),+

TCPUSER(TCPIP),+ TCPAPPL(GOMRSYL),+ TCPADDR(000.000.000.000),+ TCPPORT(00514),+ PARM(LXSYSLOG03330417UTF8)

* DEFTCPA NAME(LNXSYSL2),+

TCPUSER(TCPIP),+ TCPAPPL(GOMRSYL),+ TCPADDR(000.000.000.000),+ TCPPORT(00515),+ PARM(LXSYSLG203330417UTF8)

� Update TCP/IP configuration to allow Operations Man ager to listen for UDP traffic on the specified port(s)

– Ports 514 and 515 used here

� Update the Linux guest to send its syslog data to t he IP address and port of your z/VM system

162

IBM Software


Scenario 10:Create a Central Operations Console on One z/VM System

� Use Operations Manager to watch for error, warning, fatal messages on service machine consoles

– DIRMAINT, TCP/IP, RACF, etc.

– Linux guests

– Linux syslog

� Route these messages to a central operations consol e

� Operations staff watches operations console for sig ns of trouble

– View individual service machine consoles for more details when needed

163

IBM Software

© 2013, 2014 IBM CorporationOperational Monitoring and Automation Scenarios for a z/VM Cluster and Linux on z Systems Guests

Creating a Central Console on One z/VM System

164

z/VM

OPERATOR

Message 1bMessage 2bMount Message 3b…

DIRMAINT

Message 1bAbend Message 2bMessage 3b…

LINUX01

Urgent Message 1bMessage 2bMessage 3b…

OPMGRC1

Urgent Message 1aAbend Message 2bMount Message 3aUrgent Message 1bAbend Message 2aMount Message 3b

CP MSGNOH

DEMOADMN

Operations Mgr

(OPMGRM1)

CP SECUSER

IBM Software



� From an authorized z/VM user ID, put “abend”, “fatal”, and error messages on DIRMAINT console

msgnoh dirmaint this is a test abend message

msgnoh dirmaint this is a fake fatal message

msgnoh dirmaint DMSxxxxxxxE here is a made-up CMS e rror msg

� View the “Operations Console” to see the messagesgomcmd opmgrm1 viewcon user(oper8)

� Note the fatal message is red and abend message is highlighted and will be held when other messages come in

165

IBM Software



� From another user ID, run an EXEC to send multiple messages to the Operations Console

lotsmsgs

� View the “Operations Console” to see the messagesgomcmd opmgrm1 viewcon user(oper8)

� Watch the scrolling, held messages, etc.

166

IBM Software


IBM Software


Scenario 10: How Do You Do That?Console rules in Operations Manager:*

DEFRULE NAME(ABEND),+

MATCH(*abend*),+

EXUSER(OPER8),+

ACTION(MSGOPER8)

*

DEFRULE NAME(FATAL),+

MATCH(*fatal*),+

EXUSER(OPER8),+

ACTION(MSGOPER8)

*

DEFRULE NAME(EMSGS),+

MATCH(DMS*E),+

MCOL(001:011),+

EXUSER(OPER8),+

ACTION(MSGOPER8)

172

Action in Operations Manager:

*

DEFACTN NAME(MSGOPER8),+

COMMAND(CP MSGNOH OPER8 &U : &T),+

OUTPUT(LOG),+

ENV(LVM)

IBM Software


Scenario 10: How Do You Do That?Console rules in Operations Manager:

*

DEFRULE NAME(ABENDHLT),+

MATCH(*abend*),+

USER(OPER8),+

ACTION(HLTHOLD)

*

DEFRULE NAME(FATALRED),+

MATCH(*fatal*),+

USER(OPER8),+

ACTION(RED)

173

Actions in Operations Manager:

*

DEFACTN NAME(HLTHOLD),+

INPUT(AHI,HLD)

*

DEFACTN NAME(HILITE),+

INPUT(AHI)

*

DEFACTN NAME(RED),+

INPUT(CRE)

IBM Software


Scenario 11a:Create a Central Operations Console across multiple z/VM systems in an SSI cluster – Includes relocation of Linux and CMS guests

� Use Operations Manager to watch for error, warning, fatal messages on service machine consoles on one or more systems in an SSI cluster

– OPERATOR, DIRMAINT, TCPIP, RACF, etc.

– Linux guests

– Linux syslog

� Route these messages to a central operations consol e on one of the z/VM systems

� Operations staff watches one operations console for signs of trouble across multiple z/VM systems


174

IBM Software


Creating a Central Console Across Multiple Members of SSI Cluster

175

z/VM A z/VM B

OPERATOR

OP Message 1bOP Message 2bOP Mount Message 3b…

DIRMAINT

DIR Message 1bDIR Abend Msg 2bDIR Message 3b…

LINUX02

LX2 Urgent Msg 1bLX2 Message 2bLX2 Message 3b…

OPERSSI


CP MSGNOH

Single Configuration Users: LINUX01, LINUX02, OPERSSI, DEMOADMN Multiconfiguration (IDENTITY) Users: OPERATOR, DIRMAINT, OPMGRM1

DEMOADMN

OPERATOR


DIRMAINT


LINUX01


Operations Mgr(OPMGRM1)

CP SECUSER

Operations Mgr(OPMGRM1)CP MSGNOH

CP SECUSER

DEMOADMN

IBM Software



� On System B (TEST7SSI), view the “Operations Consol e” (user ID OPERSSI)


� On System A (TEST7SSI), find a Linux guest running disconnected locally and relocate it

q names

VMRELOCATE MOVE USER RHEL5G TO TESTCSSI

� On System B (TEST7SSI), prepare for planned shutdow n by relocating the central operations console (OPERSSI)

VMRELOCATE MOVE USER OPERSSI TO TESTCSSI

� Note the messages received on OPERSSI on TEST7SSI f rom OPERATOR on both TESTCSSI and TEST7SSI indicating RHEL5G was relocated

� Note the message received on OPERSSI on TESTCSSI indicating OPERSSI has been relocated

176

IBM Software


IBM Software



Event monitor in Operations Manager:*

*** Notify OPERSSI console when relocations started

DEFEMON NAME(RELOC),+

TYPE(9,10),+

ACTION(RELOC)

183


DEFACTN NAME(RELOC),+

COMMAND(EXEC MSG2OPER &u &3 &4 &5 junk),+

ENV(LVM)

IBM Software



MSG2OPER EXEC (excerpts):

/* Send a message to a central console OPERSSI for SSI cluster */

/* */

trace r

Address Command

Parse arg userid euser event sourcesys msgtext

/* Get local TCP/IP hostname */

parse value Search_TCPIP_Data("hostname") with getrc tcphostname .

if getrc > 4 then tcphostname = "unknown_host_name"

if userid = '_GOMEMON' then

do

if event = 9 then

msgtext = 'Outbound relocation for' euser 'on' sourcesys 'started'

else

msgtext = 'Inbound relocation for' euser 'on' sourcesys 'started'

'CP MSGNOH OPERSSI AT TEST7SSI From' tcphostname ':' msgtext

end

184

IBM Software


Scenario 11b:Create a Central Operations Console across multiple z/VM systems that are not in an SSI cluster

� Use Operations Manager to watch for error, warning, fatal messages on service machine consoles on one or more systems

– OPERATOR, DIRMAINT, TCP/IP, RACF, etc.

– Linux guests

– Linux syslog

� Route these messages to a central operations consol e on one of the z/VM systems

� Operations staff watches one operations console for signs of trouble across multiple z/VM systems


185

IBM Software


Creating a Central Console Across Multiple LPARS

186

z/VM A z/VM B

OPERATOR


DIRMAINT


LINUX02


OPMGRC1


IP (Ops Mgr)

DEMOADMN

OPERATOR


DIRMAINT


LINUX02


Operations Mgr(OPMGRM1)

CP SECUSER

Operations Mgr(OPMGRM1)CP MSGNOH

CP SECUSER

IBM Software


IBM Software



� On System A (DEM1ZVM) put an “error” message on the OPERATOR console

– Must contain the text “remote error”msgnoh operator here is a remote error message

� View the “Operations Console” (user ID OPMGRC1) on System B (ZVMV5R40) to see the message


� Note the message received on OPMGRC1 on ZVMV5R40 from OPERATOR on DEM1ZVM

189

IBM Software


IBM Software


Scenario 11b: How Do You Do That?Console rule in Operations Manager on System A:*

DEFRULE NAME(OPERMSGS),+

MATCH(*remote error*),+

USER(OPERATOR),+

ACTION(MSG2GBRG)

193

Action in Operations Manager on System A:

*

DEFACTN NAME(MSG2GBRG),+

COMMAND(EXEC MSG2OPS OPMGRC1 From &u on DEM1ZVM: &t),+

OUTPUT(LOG),+

ENV(LVM)

IBM Software


Scenario 11b: How Do You Do That?MSG2OPS EXEC on System A:

/* Send a message to a console in Ops Mgr on anoth er system */

/* */

trace r

Address Command

Parse arg cons_user msgtext

'PIPE var msgtext | > TEMP NOTE A'

'EXEC GOMRSIF TEMP NOTE A 9.82.24.129 63000' cons_u ser

Exit

194

IP address of System B

Central Console (OPMGRC1)

IBM Software


Scenario 11b: How Do You Do That?TCP/IP listener definition in Operations Manager on System B:*

DEFTCPA NAME(TESTDATA),+

TCPUSER(TCPIP),+

TCPAPPL(GOMRSIF),+

TCPADDR(000.000.000.000),+

TCPPORT(63000)

� May also need to update TCP/IP on System B to allow Operations Manager to listen on port 63000

� Can alternatively use TELL (instead of GOMRSIF) to send messages from System A to System B, but requires RS CS

195

IBM Software


Scenario 12Integration with OMEGAMON XE on z/VM and Linux

� Use Operations Manager to take action based on a tr iggered situation in OMEGAMON XE on z/VM and Linux

� Virtual CPU consumption is high for a Linux guest

� OMEGAMON detects the situation, creates an event, a nd sends message to Operations Manager

� Action is triggered by a rule in Operations Manager

� Operations Manager checks SHARE status of guest and issues CP commands to tune the guest

– SET QUICKDSP

– SET SHARE

� Event is resolved in OMEGAMON when virtual CPU consumption of guest is back down

196

IBM Software


OMEGAMON XE and Operations Manager for z/VM

197

Linux Guest Linux GuestLinux Guest

Web browser (TEP)

Perf

ToolkitVM Agent

TMS

Linux Agent Linux Agent

Web ServerDatabase

Server

z System

Operations

Manager

5

Process Flow

*MONITORz/VM 1

2

22

VM Cmd Proc

3

3

4

IBM Software



� Create and start an application on a Linux guest th at uses more than 20% of virtual CPU– HOG command on our demo system

� Updates to Tivoli Enterprise Portal– z/VM CPU graph shows guest CPU % as it runs the application– Event pops up on situation event console to say higher than 20%

� Use Operations Manager to watch z/VM user console us ed by OMEGAMON– Message receive from OMEGAMON to address high CPU on the

guest– Message from Operations Manager indicating action is triggered

� Updates on Tivoli Enterprise Portal– CPU used by that guest decreases below 20%– Event closed (removed from the event console)

198

IBM Software


IBM Software



Rules in Operations Manager:** Adjust SHARE of Linux guest if CPU usage is too h igh

* Watch for message from OMEGAMON

DEFRULE NAME(GUSTCPU),+

MATCH(*NEEDS CPU PRIORITY*),+

ACTION(GUESTCPU)

** Highlight message from OMEGAMON and call EXEC to check and adjust

* SHARE of Linux guest

DEFACTN NAME(GUESTCPU),+

INPUT(AHI),+

NEXTACTN(GUSTCPUB)

*

DEFACTN NAME(GUSTCPUB),+

COMMAND(EXEC VCPU &4),+

ENV(LVM),+

OUTPUT(LOG)

204

IBM Software


Scenario 12: Detailed StepsOMEGAMON Configuration

205

IBM Software


Scenario 12: Detailed StepsOMEGAMON Configuration

206

IBM Software


Scenario 13:Monitor Service Machines for LOGOFF Status – and AUTOLOG them

� Monitor specific service machines to make sure they stay logged on

– Demo will monitor TSTADMN2 user ID

• Could monitor a group of user IDs

– If it changes from logged on to logged off status, then restart it

� Dynamically pass the user ID to the action

– Re-use action for multiple monitors or user IDs

207

IBM Software


IBM Software




DEFEMON NAME(ADMIN2),+

TYPE(1),+

USER(TSTADMN2),+

ACTION(AUTOLOG1)

*

DEFACTN NAME(AUTOLOG1),+

COMMAND(CP SLEEP 3 SEC),+

NEXTACTN(AUTOLOG2),+

OUTPUT(LOG),+

ENV(OPMGRS1)

*

DEFACTN NAME(AUTOLOG2),+

COMMAND(CP XAUTOLOG &3),+

OUTPUT(LOG),+

ENV(OPMGRS1)

212

IBM Software


Scenario 14:Monitor Page Space – Send Email if Full

� Operations Manager monitors the page space usage (percent full)

– For demo purposes, page space monitor is currently defined but suspended (not active)

– We’ll dynamically resume (re-activate) the page space monitor

– Demo monitor requires the page space be only 0% full


� Automatically send an e-mail to someone who can eva luate and take action

� For demo purposes, suspend (de-activate) the page s pace monitor when complete

213

IBM Software



� From an authorized VM user ID, see the page space u sage:q alloc page

� From a user ID with Operations Manager privileges:gomcmd opmgrm1 resume page(pgfull)

� Check the Operations Manager log to see the spool m onitor triggered:


� Check the inbox of the appropriate person to see th e email

� From a user ID with Operations Manager privileges:gomcmd opmgrm1 suspend page(pgfull)

214

IBM Software


IBM Software



� Console rule and action in Operations Manager: *

DEFPMON NAME(PGFULL),+

USAGE(010-100),+

INTERVAL(1),+

LIMIT(3,3600),+

ACTION(PAGEMAIL)

*

SUSPEND PAGE(PGFULL)

*

DEFACTN NAME(PAGEMAIL),+

COMMAND(EXEC SMTPPG tld1 at us.ibm.com &4),+

OUTPUT(LOG),+

ENV(LVM)

218

IBM Software



219

SMTPPG EXEC (excerpts)/* */Parse arg mail_user dummyat mail_node pgpct

errtext = ‘Page space is' pgpct‘% full on z/VM syst em‘

/* Get local TCP/IP hostname */ parse value Search_TCPIP_Data("hostname") with getr c tcphostname . if getrc > 4 then tcphostname = "unknown_host_name" parse value Search_TCPIP_Data("domainorigin") with getrc tcpdomain . if getrc > 4 then tcpdomain = "unknown_domain_name"

fqdomain_name = tcphostname'.'tcpdomain

/* Construct the e-mail */line.1 = 'OPTIONS: NOACK LOG SHORT NONOTEBO OK ALL CLASS A' line.2 = 'Date: ' Date() ',' Time() line.3 = 'From: Operations Manager for z/VM' line.4 = 'To: ' mail_user 'at' mail_node line.5 = 'Subject: ' errtext 'on' fqdomain_nameline.6 = ‘DO NOT REPLY - This e-mail was generated b y an automated service machine’Line.7 = ‘ ‘line.8 = msgtext line.0 = 8

'PIPE stem line. | > TEMP NOTE A'

'EXEC SENDFILE TEMP NOTE A (NOTE SMTP'

IBM Software


Scenario 15:Monitor SSI Connectivity between Two Members of a Cluster

� Create a schedule to query ISLINKs between two members of a cluster

� If less than 4 links up, send message to consolidated SSI console (OPERSSI)

– For demo purposes, we’ll dynamically deactivate a link then reactivate it when done

220

IBM Software



� From an authorized VM user ID, see the currently av ailable ISLINKs:

q islink node testcssi

� Deactivate one of the links:deactivate islink 0d01

� Using Operations Manager, view the central operatio ns console to see the alert:


� Schedule is triggered every 2 minutes, so wait 2 mi nutes and see the messages again

� Reactivate the link:activate islink 0d01

221

IBM Software



� Schedule and action in Operations Manager: *** Check every 10 minutes for any IS links being d own

DEFSCHD NAME(ISLINK1),+

EVERY(00:02),+

ACTION(QISLINK),+

PARM(TESTCSSI)

*

DEFACTN NAME(QISLINK),+

COMMAND(EXEC QISLINK TEST7SSI &p),+

ENV(SVM)

222

IBM Software



� QISLINK EXEC: /* Find the number of IS Links available to anothe r node */

/* If less than 4, then send message to OPERSSI */

trace o

Address command

Parse Arg thisnode othernode

'PIPE CP QUERY ISLINK NODE' othernode '| find _____ State:____Up| COUNT LINES | VAR numlinks'

If numlinks < 4

Then 'CP MSGNOH OPERSSI AT TEST7SSI From' thisnode' : Number of ISLINKs to' othernode 'is' numlinks

Exit 0

223

IBM Software


Scenario 16:Suppress Passwords on Linux Consoles

� TN3270 login to Linux guest displays password

– Password on separate line from password prompt

– Password captured in console and viewable in Operations Manager VIEWCON

� Use a rule in Operations Manager to suppress the password

– I.e. the line following the “password:” prompt

� Can be expanded to suppress multiple lines following matching text

224

IBM Software



� Use Operations Manager to view the console of a Linux guest:

gomcmd opmgrm1 viewcon user(omeglnx1)

� Enter the login command:login root

� Enter the password

– Note that it’s not displayed

225

IBM Software



� Rule and action in Operations Manager: *

* Change password prompt to red.

* Suppress the password when logging onto OMEGLNX1.

* Have to suppress next 2 lines to include the line Ops Mgr adds

* indicating the user entered a "command"

*

DEFRULE NAME(OMEGPW),+

MATCH(Password:*),+

USER(OMEGLNX1),+

ACTION(SUPPW),+

SUPNEXT(2)

*

DEFACTN NAME(SUPPW),+

INPUT(CRE)

226

IBM Software


Scenario 17:Autolog a Linux Guest and Send Message if Doesn’t Start Successfully

� Define a schedule and action to start a Linux guest

� Define a rule looking for the application specific message indicating up and ready for work

� Define an idle monitor for the above rule

– If “up and ready” message is not found within 1 minute, then send message to central console

� Idle monitor is suspended until schedule is trigger ed

– Before autologging the Linux guest, automatically resume idle monitor

� Idle monitor is automatically suspended again once it is triggered

227

IBM Software



� View the configuration file to see the action that will be “scheduled”, plus the rules and monitors

x tracy config

� Run the action that starts the guest (and monitors)gomcmd opmgrm1 run action(strtlnx1)

� View the console of LNXTEST to see that it gets autologged

gomcmd opmgrm1 viewcon user(lnxtest)

� View the central console of OPERSSI to see the message that the guest did not start successfully


228

IBM Software


IBM Software



Schedule and action in Operations Manager:*DEFSCHD NAME(STARTLNX),+

* WHEN(00:01),+

* ACTION(STRTLNX1)

*

DEFACTN NAME(STRTLNX1),+

COMMAND('RESUME IDLE(NOLOGON)'),+

NEXTACTN(STRTLNX2),+

ENV(GOM)

*

DEFACTN NAME(STRTLNX2),+

COMMAND(CP XAUTOLOG LNXTEST),+

ENV(SVM)

233

IBM Software



�Watch for successful startup of Linux guest

� If successful take no actionDEFRULE NAME(LNXLOGON),+

MATCH(*LNXTEST successfully started*),+

USER(LNXTEST),+

ACTION(NOACT)

*

DEFACTN NAME(NOACT)

234

IBM Software



� If Linux doesn’t start successfully send message to central console and suspend monitor:

DEFIMON NAME(NOLOGON),+

RULE(LNXLOGON),+

OCCUR(1,1),+

ACTION(MSG2SSI2),+

PARM(LNXTEST)

*

SUSPEND IDLE(NOLOGON)

DEFACTN NAME(MSG2SSI2),+

COMMAND(CP MSGNOH OPERSSI &p did not successfully complete startup),+

NEXTACTN(SUSPIDLE),+

ENV(SVM)

*

DEFACTN NAME(SUSPIDLE),+

COMMAND('SUSPEND IDLE(NOLOGON)'),+

ENV(GOM)

235

operational monitoring and automation for zvm zlinux 20150303 › share › 124 ›...

Documents