system automation for z/os: customizing and …...syslog pr ocessing ..... . 190 system log failur e...

System Automation for z/OS

Customizing and ProgrammingVersion 3 Release 5

SC34-2715-00

IBM

NoteBefore using this information and the product it supports, read the information in Appendix G, “Notices,” on page 311.

Edition Notes

This edition applies to IBM Tivoli System Automation for z/OS (Program Number 5698-SA3) Version 3, Release 5,an IBM licensed program, and to all subsequent releases and modifications until otherwise indicated in neweditions.

This edition replaces SC34-2644-00.

© Copyright IBM Corporation 1996, 2014.US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contractwith IBM Corp.

Contents

Figures . . . . . . . . . . . . . . . ix

Tables . . . . . . . . . . . . . . . xi

Accessibility . . . . . . . . . . . . xiiiUsing assistive technologies . . . . . . . . xiiiKeyboard navigation of the user interface . . . . xiii

Dotted decimal syntax diagrams. . . . xv

How to send your comments to IBM xvii

About this publication . . . . . . . . xixWho Should Use This Publication. . . . . . . xixPrerequisites . . . . . . . . . . . . . . xixWhere to Find More Information . . . . . . . xix

The System Automation for z/OS Library . . . xixRelated Product Information. . . . . . . . xx

Summary of Changes for SC34-2715-00 . . . . . xxNew Information . . . . . . . . . . . xxChanged Information . . . . . . . . . . xxiMoved Information . . . . . . . . . . xxiDeleted Information . . . . . . . . . . xxi

Chapter 1. Adding a New Application toAutomation . . . . . . . . . . . . . 1Preparation Before Automating an Application . . . 1

Address Space properties . . . . . . . . . 1Step 1 - Application Start . . . . . . . . . 1Step 2 - Application Stop . . . . . . . . . 2Step 3 - Application Events . . . . . . . . 3Step 4 - Application Monitoring . . . . . . . 4Outstanding Reply Processing . . . . . . . 5Topology . . . . . . . . . . . . . . 5

Adding the Application to Automation . . . . . 5Define an Application Policy Object . . . . . 5Using Automated Discovery to Define ApplicationPolicy Objects . . . . . . . . . . . . . 6Build New Automation Configuration Files . . . 6

Chapter 2. Creating AutomationProcedures . . . . . . . . . . . . . 7How Automation Procedures Are Called . . . . . 7How Automation Procedures Are Structured . . . 8

Performing Initialization Processing . . . . . 9Determining whether Automation Is Allowed . . 9Performing Automation Processing . . . . . 10

How to Make Your Automation Procedures Generic 14Processor Operations Commands . . . . . . 15

Developing Messages for Your AutomationProcedures . . . . . . . . . . . . . . 15

Example AOCMSG Call . . . . . . . . . 16Example Automation Procedure . . . . . . . 16

Notes on the Automation Procedure Example . . 18Installing Your Automation Procedures . . . . . 19Testing and Debugging Automation Procedures . . 19

The Assist Mode Facility . . . . . . . . . 19Using Assist Mode to Test AutomationProcedures . . . . . . . . . . . . . 19Using AOCTRACE to Trace AutomationProcedure Processing . . . . . . . . . . 20NetView Testing and Debugging Facilities . . . 21Where to Find More Testing Information . . . 22

Coding Your Own Information in the AutomationStatus File . . . . . . . . . . . . . . . 22Programming Recommendations . . . . . . . 22Global Variable Names . . . . . . . . . . 23

Chapter 3. Adding a Message toAutomation . . . . . . . . . . . . . 25Conceptual Overview . . . . . . . . . . . 25Defining Actions for Messages . . . . . . . . 25

Defining CMD or REP Actions . . . . . . . 26Defining AT Actions . . . . . . . . . . 26Defining Message Overrides . . . . . . . . 28

Extended Status Command Support . . . . . . 28Policy Definitions . . . . . . . . . . . 29

Defining Entries for the Message Revision Table . . 30Build . . . . . . . . . . . . . . . . 31AT and MRT Build Concept . . . . . . . . . 31Load . . . . . . . . . . . . . . . . 32

Listings. . . . . . . . . . . . . . . 32A Guide to SA z/OS Automation Tables . . . . 33

NetView Automation Table Structure . . . . . 33Integrating Automation Tables . . . . . . . 34Generic Automation Table Statements . . . . 35

System Operations Automation Flow . . . . . . 35Inheritance Rules for Classes . . . . . . . . 37

Define Application Information . . . . . . . 37Define Relationships . . . . . . . . . . 37Define Application Messages and User Data . . 37Define Startup Procedures . . . . . . . . 37Define Shutdown Procedures . . . . . . . 38Define Error Thresholds . . . . . . . . . 38Define IMS Subsystem-Specific Data . . . . . 38

Chapter 4. Monitoring Applications . . 41Observed Status Monitoring . . . . . . . . . 41Health Monitoring . . . . . . . . . . . . 42

Overview . . . . . . . . . . . . . . 42Monitor Resource Commands . . . . . . . 43Writing a Recovery Routine . . . . . . . . 44Active Health Monitoring . . . . . . . . 45Passive, Event-Based Health Monitoring . . . . 46Programming Techniques . . . . . . . . . 48

Health Monitoring using OMEGAMON . . . . . 50Overview . . . . . . . . . . . . . . 50Assumptions . . . . . . . . . . . . . 51

© Copyright IBM Corp. 1996, 2014 iii

OMEGAMON Interaction . . . . . . . . 52Health Monitoring Based on OMEGAMONExceptions. . . . . . . . . . . . . . 55Health Monitoring Based on OMEGAMON XESituations . . . . . . . . . . . . . . 57

Health Monitoring using CICSPlex SM . . . . . 60Component Overview . . . . . . . . . . 60Creating an Application to Manage the VOST . . 61Defining the Monitor Resources . . . . . . 61

Monitoring JES3 Components . . . . . . . . 61AOFRJ3MN Routine . . . . . . . . . . 62AOFRJ3RC Routine. . . . . . . . . . . 64

JES2 Spool Monitoring. . . . . . . . . . . 65DB2 Connection Monitoring . . . . . . . . . 65IMS Component Monitoring . . . . . . . . . 66

Chapter 5. Joblog Monitoring . . . . . 67Overview . . . . . . . . . . . . . . . 67Customization . . . . . . . . . . . . . 68Status Information . . . . . . . . . . . . 73

Chapter 6. Alert-Based Notification . . 75Overview . . . . . . . . . . . . . . . 75Communication Flow . . . . . . . . . . . 75Enabling Alerting . . . . . . . . . . . . 76

Setup in SA z/OS . . . . . . . . . . . 76INGALERT Command. . . . . . . . . . . 80

Chapter 7. Availability and RecoveryTime Reporting . . . . . . . . . . . 83Overview . . . . . . . . . . . . . . . 83Resource Lifecycle . . . . . . . . . . . . 83Layout of the SMF Record . . . . . . . . . 84Enabling SMF Records. . . . . . . . . . . 85The INGPUSMF Utility . . . . . . . . . . 86

Output . . . . . . . . . . . . . . . 86The INGPUSMF Utility JCL . . . . . . . . 86Return Codes. . . . . . . . . . . . . 87

Writing the SMF Report to DB2. . . . . . . . 88Customization . . . . . . . . . . . . 88Output . . . . . . . . . . . . . . . 89

Chapter 8. Automating ProcessorOperations-Controlled Resources . . . 91Automating Processor Operations Resources ofz/OS Target Systems Using Proxy Definitions . . . 91

Concept . . . . . . . . . . . . . . 91Customizing Automation for Proxy Resources . . 92Preparing Message Automation. . . . . . . 93

Automating Linux Console Messages . . . . . . 94The Linux Console Connection to NetView . . . 94Linux Console Automation with Mixed CaseCharacter Data . . . . . . . . . . . . 94Security Considerations . . . . . . . . . 94Restrictions and Limitations . . . . . . . . 95

How to Add a Processor Operations Message toAutomation . . . . . . . . . . . . . . 95

Messages Issued by a Processor OperationsTarget System . . . . . . . . . . . . 95Building the New Automation Definitions . . . 99

Loading the Changed Automation Environment 99Using Pipes and ISQCCMD for Synchronous HWCommands . . . . . . . . . . . . . . 100Automating Asynchronous Hardware Commandswith ISQCCMD and PIPES . . . . . . . . . 101VM Second Level Systems Support . . . . . . 102

Guest Target Systems. . . . . . . . . . 102Customizing Target Systems . . . . . . . 103

Chapter 9. Automating USSResources. . . . . . . . . . . . . 107Integration of z/OS UNIX System Services . . . 107

Infrastructure Overview . . . . . . . . . 107Setting Up z/OS UNIX Automation . . . . . . 108

Customization of z/OS UNIX Resources . . . 108Example: sshd . . . . . . . . . . . . 113

Hints and Tips . . . . . . . . . . . . . 117Trapping UNIX syslogd Messages . . . . . 118Debugging . . . . . . . . . . . . . 118

Chapter 10. Command Receiver . . . 119Setting Up the Command Receiver . . . . . . 119Setting Up TSO/Batch Environment . . . . . . 119Defining Command Receiver as a SubsystemAutomated by SA z/OS. . . . . . . . . . 120Defining Command Work Tasks Used by theCommand Receiver . . . . . . . . . . . 121Starting and Stopping the Command Receiver . . 121Submitting NetView Commands from a Batch Job 121

Sample Batch Job JCL . . . . . . . . . 121Command Statement Syntax . . . . . . . 122Executing a Command on a Different NetView 123JCL for the Batch Command Interface . . . . 123

Chapter 11. Enabling Relational DataServices (RDS). . . . . . . . . . . 127Enable/Disable Persistent Relational Data Tables 127Import System Automation Resources . . . . . 127Regular Snapshot . . . . . . . . . . . . 128RDS Initialization . . . . . . . . . . . . 128Use RDS for TSO . . . . . . . . . . . . 128

Setup Security Profiles . . . . . . . . . 129RDS Table Editor . . . . . . . . . . . . 130

Customize the TSO Environment . . . . . . 130Define a RDS Working Data Set forViewing/Editing under TSO . . . . . . . 130

Chapter 12. Enabling SysplexAutomation . . . . . . . . . . . . 133Sysplex Functions . . . . . . . . . . . . 133

Managing Couple Data Sets . . . . . . . 133Managing the System Logger . . . . . . . 134Managing Coupling Facilities . . . . . . . 135Recovery Actions . . . . . . . . . . . 137Hardware Validation . . . . . . . . . . 144

Enabling Hardware-Related Automation . . . . 146Step 1: Defining the Processor . . . . . . . 146Step 2: Using the Policy Item PROCESSORINFO . . . . . . . . . . . . . . . 146

iv System Automation for z/OS: Customizing and Programming

||

||||

||

Step 3: Defining Logical Partitions . . . . . 146Step 4: Defining the System . . . . . . . 146Step 5: Connecting the System to the Processor 146Step 6: Defining Logical Sysplexes . . . . . 147Step 7: Defining the Physical Sysplex . . . . 147

Enabling Continuous Availability of Couple DataSets . . . . . . . . . . . . . . . . 147Enabling WTO(R) Buffer Shortage Recovery . . . 148Enabling System Removal . . . . . . . . . 148

Step 1: Defining the Processor and System. . . 149Step 2: Defining the Application withApplication Type IMAGE . . . . . . . . 149Step 3: Defining an Application Group . . . . 149Step 4: Automating IXC102A and IXC402DMessages . . . . . . . . . . . . . . 149Step 5: Verify Automation table entries . . . . 151

Enabling Long Running Enqueues (ENQs) . . . 151Step 1: Defining Resources . . . . . . . . 151Step 2: Making Job/ASID Definitions . . . . 151Step 3: Defining IEADMCxx Symbols . . . . 152Step 4: Defining Commands . . . . . . . 152Step 5: Defining Snapshot Intervals . . . . . 152

Enabling Auxiliary Storage Shortage Recovery . . 152Step 1: Defining the Local Page Data Set . . . 152Step 2: Defining the Handling of Jobs . . . . 152

Defining Common Automation Items . . . . . 153Customizing the System to Use the Functions . . 153

Additional Automation Operator IDs . . . . 153Switching Sysplex Functions On and Off . . . 153

Chapter 13. Automating Networks . . 155Automation Network Definition Process . . . . 155

Defining an SDF Focal Point System. . . . . 155Defining Gateway Sessions . . . . . . . . 156

Defining Automatically-Initiated TAF FullscreenSessions . . . . . . . . . . . . . . . 156

Chapter 14. Defining a VTAMApplication to SA z/OS . . . . . . . 159Registering VTAM Application Subsystems withSA z/OS Recovery . . . . . . . . . . . 159

Chapter 15. Shutting Down z/OSsystems in a GDPS Environment . . . 161Example Definition . . . . . . . . . . . 162INGREQ Considerations. . . . . . . . . . 162

Chapter 16. WTOR Processing . . . . 165Process Flow of WTORs . . . . . . . . . . 165

Actions in Response to Incoming WTORs . . . 166Customizing how WTORs Are Stored bySA z/OS . . . . . . . . . . . . . . 166Processing of Primary WTORs. . . . . . . 167Usage Notes. . . . . . . . . . . . . 168

Chapter 17. SA z/OS User Exits . . . 169Initialization Exits . . . . . . . . . . . . 169

AOFEXDEF . . . . . . . . . . . . . 170AOFEXI01 . . . . . . . . . . . . . 171

AOFEXI02 . . . . . . . . . . . . . 171AOFEXI03 . . . . . . . . . . . . . 171AOFEXI04 . . . . . . . . . . . . . 171AOFEXI05 . . . . . . . . . . . . . 171AOFEXI06 . . . . . . . . . . . . . 172AOFEXINT . . . . . . . . . . . . . 172Environmental Setup Exits . . . . . . . . 172

Static Exits . . . . . . . . . . . . . . 173AOFEXSTA . . . . . . . . . . . . . 173AOFEXX02 . . . . . . . . . . . . . 174AOFEXX03 . . . . . . . . . . . . . 174AOFEXX04 . . . . . . . . . . . . . 175AOFEXX05 . . . . . . . . . . . . . 175AOFEXX15 . . . . . . . . . . . . . 175

Flag Exits. . . . . . . . . . . . . . . 175Parameters . . . . . . . . . . . . . 177Task Global Variables. . . . . . . . . . 178Return Codes . . . . . . . . . . . . 178

Customization Dialog Exits . . . . . . . . . 178User Exits for BUILD Processing . . . . . . 179User Exits for COPY Processing . . . . . . 180User Exits for DELETE Processing . . . . . 180User Exits for CONVERT Processing . . . . 181User Exits for RENAME and IMPORT Functions 181User Exits for Other Functions . . . . . . 182Invocation of Customization Dialog Exits . . . 182

Command Exits . . . . . . . . . . . . 182AOFEXC00 . . . . . . . . . . . . . 183AOFEXC01 . . . . . . . . . . . . . 183AOFEXC02 . . . . . . . . . . . . . 183AOFEXC03 . . . . . . . . . . . . . 183AOFEXC04 . . . . . . . . . . . . . 183AOFEXC05 . . . . . . . . . . . . . 183AOFEXC06 . . . . . . . . . . . . . 184AOFEXC07 . . . . . . . . . . . . . 184AOFEXC08 . . . . . . . . . . . . . 184AOFEXC09 . . . . . . . . . . . . . 184AOFEXC10 . . . . . . . . . . . . . 184AOFEXC11 . . . . . . . . . . . . . 184AOFEXC13 . . . . . . . . . . . . . 185AOFEXC14 . . . . . . . . . . . . . 185AOFEXC15 . . . . . . . . . . . . . 185AOFEXC16 . . . . . . . . . . . . . 185AOFEXC17 . . . . . . . . . . . . . 185AOFEXC18 . . . . . . . . . . . . . 185AOFEXC19 . . . . . . . . . . . . . 186AOFEXC20 . . . . . . . . . . . . . 186AOFEXC21 . . . . . . . . . . . . . 186AOFEXC22 . . . . . . . . . . . . . 187AOFEXC23 . . . . . . . . . . . . . 187AOFEXC24 . . . . . . . . . . . . . 187AOFEXC25 . . . . . . . . . . . . . 187

Pseudo-Exits . . . . . . . . . . . . . 187Automation Control File Reload Permission Exit 187Automation Control File Reload Action Exit . . 188Subsystem Up at Initialization Commands . . 188

Testing Exits. . . . . . . . . . . . . . 188

Chapter 18. Automation Solutions . . 189LOGREC Data Set Processing . . . . . . . . 190SMF Data Set Processing . . . . . . . . . 190

Contents v

||

SYSLOG Processing . . . . . . . . . . . 190System Log Failure Recovery . . . . . . . . 191SVC Dump Processing . . . . . . . . . . 191Deletion of Processed WTORs from the Display 191AMRF Buffer Shortage Processing . . . . . . 191Drain Processing Prior to JES2 Shutdown . . . . 192IMS Transaction Recovery . . . . . . . . . 192Looping Address Space Suppression. . . . . . 192

Preparation . . . . . . . . . . . . . 193Copy the Sample Policy . . . . . . . . . 193Customizing your SOAP Server . . . . . . 193Configuring your APL . . . . . . . . . 194Linkage . . . . . . . . . . . . . . 197Build . . . . . . . . . . . . . . . 197Prime the Credentials . . . . . . . . . 197Activate the Monitor . . . . . . . . . . 198Verify the Installation. . . . . . . . . . 198

AOFRSA01 . . . . . . . . . . . . . . 198AOFRSA02 . . . . . . . . . . . . . . 199AOFRSA03 . . . . . . . . . . . . . . 201AOFRSA08 . . . . . . . . . . . . . . 203AOFRSA0C . . . . . . . . . . . . . . 205AOFRSA0E . . . . . . . . . . . . . . 208AOFRSA0G . . . . . . . . . . . . . . 209AOFRSD07 . . . . . . . . . . . . . . 211AOFRSD09 . . . . . . . . . . . . . . 212AOFRSD0F . . . . . . . . . . . . . . 213AOFRSD0G . . . . . . . . . . . . . . 215AOFRSD0H . . . . . . . . . . . . . . 216HASP099 . . . . . . . . . . . . . . . 218INGRMJSP . . . . . . . . . . . . . . 219INGRCJSP (AOFRSD01) . . . . . . . . . . 220INGRTAPE . . . . . . . . . . . . . . 222INGRX711 . . . . . . . . . . . . . . 223INGRX740 . . . . . . . . . . . . . . 223INGRYSHU . . . . . . . . . . . . . . 225

Chapter 19. Automated SystemResource Discovery . . . . . . . . 227Disclaimer . . . . . . . . . . . . . . 227Components Overview . . . . . . . . . . 228Overview of Using the Automated SystemResource Discovery Process . . . . . . . . 230

Step 1: Using The Discovery Engine . . . . . 230Step 2: Building the Automation Policy . . . . 232

Extending Automated Modelling . . . . . . . 236Mapping Files . . . . . . . . . . . . 236

Finalizing the Target Policy. . . . . . . . . 237Creating and preparing a Reference Policy . . 237Importing from the Reference Policy . . . . 237Avoiding multiple entries for the sameapplication . . . . . . . . . . . . . 237Changing the Name of an APL. . . . . . . 238Target Policy Ready for Build . . . . . . . 238

Building the Configuration Control Data . . . . 238Troubleshooting . . . . . . . . . . . . 239

Appendix A. Global Variables . . . . 241Read-Only Variables . . . . . . . . . . . 241Read/Write Variables. . . . . . . . . . . 242

Parameter Defaults for Commands . . . . . . 255

Appendix B. Customizing the StatusDisplay Facility . . . . . . . . . . 265Overview of the Status Display Facility. . . . . 265

How the Status Display Facility Works . . . . 265Types of SDF Panels . . . . . . . . . . 265Status Descriptors . . . . . . . . . . . 266SDF Tree Structures . . . . . . . . . . 267How Status Descriptors Affect SDF . . . . . 268How SDF Helps Operations to Focus on SpecificProblems . . . . . . . . . . . . . . 271How SDF Panels Are Defined . . . . . . . 272Dynamically Loading Tree Structure and PanelDefinition Members . . . . . . . . . . 272Using SDF for Multiple Systems . . . . . . 273SDF Components . . . . . . . . . . . 273How the SDF Task Is Started and Stopped . . 274

SDF Definition Process . . . . . . . . . . 274Step 1: Defining SDF Hierarchy . . . . . . 275Step 2: Defining SDF Panels . . . . . . . 276Step 3: (Optional) Customizing SDFInitialization Parameters . . . . . . . . . 280Step 4: (Optional) Defining SDF in theCustomization Dialog . . . . . . . . . 280

Appendix C. Coordinating SystemOperations with Automatic RestartManager . . . . . . . . . . . . . 281Defining an ARM Element Name. . . . . . . 281Defining a MOVE Group for Automatic RestartManager . . . . . . . . . . . . . . . 282

Appendix D. Message Automation . . 283Generic Synonyms: AOFMSGSY . . . . . . . 283

SA z/OS Message Presentation: AOFMSGSY 284Operator Cascades: AOFMSGSY . . . . . . 286SA z/OS Topology Manager for NMC:AOFMSGST . . . . . . . . . . . . . 289

Appendix E. TSO User Monitoring . . 291

Appendix F. Autodiscovery MapperFiles and Report Formats . . . . . . 293Mapper Files . . . . . . . . . . . . . 293

Functions. . . . . . . . . . . . . . 293Address Space Identification Mapping File (INGSMAID / INGSMAIU) . . . . . . . . 294XCF Group Identification Mapping File (INGSMGRP / INGSMGRU) . . . . . . . 295XCF Group Member Identification Mapping File(INGSMGMB / INGSMGMU) . . . . . . . 295USS Process Identification Mapping File(INGSMUID / INGSMUIU) . . . . . . . 296Policy Mapping File (INGSMPLU / INGSMPLY) 297Variable Mapping File (INGSMVRS /INGSMVRU) . . . . . . . . . . . . 298

Preloader Reports . . . . . . . . . . . . 299Summary Report . . . . . . . . . . . 300

vi System Automation for z/OS: Customizing and Programming

||||||||||||||||||||

||

XCFGROUP Report . . . . . . . . . . 301ARMGROUP Report . . . . . . . . . . 302ASDETAIL Report . . . . . . . . . . . 303EXCLUDE Report . . . . . . . . . . . 304CONSTRCT Report . . . . . . . . . . 305SYMBOLS Report . . . . . . . . . . . 306KBIMPORT Report . . . . . . . . . . 307KBMAP Report. . . . . . . . . . . . 308SUBSYS Report . . . . . . . . . . . . 309

Appendix G. Notices . . . . . . . . 311Trademarks . . . . . . . . . . . . . . 313Terms and conditions for product documentation 313

Glossary . . . . . . . . . . . . . 315

Index . . . . . . . . . . . . . . . 345

Contents vii

viii System Automation for z/OS: Customizing and Programming

Figures

1. Application Lifecycle . . . . . . . . . . 42. Automation Procedures for System Operations 83. Automation Procedures for Processor

Operations . . . . . . . . . . . . . 84. Skeleton of an Automation Procedure . . . . 155. AT Structure . . . . . . . . . . . . 336. Sample Monitor Command . . . . . . . 467. Take Action Dialog . . . . . . . . . . 598. ISPF dialog defining the joblog monitoring for

an application. . . . . . . . . . . . 699. ISPF dialog defining the joblog monitoring of

all JESMSGLG messages (1/3) . . . . . . 6910. ISPF dialog defining the joblog monitoring of

all JESMSGLG messages (2/3) . . . . . . 6911. ISPF dialog defining the automation of specific

messages . . . . . . . . . . . . . 7012. ISPF dialog defining the joblog monitoring of

specific messages . . . . . . . . . . 7013. ISPF dialog defining the joblog monitoring of

specific messages of a multi-step job . . . . 7014. ISPF dialog defining the joblog monitoring of a

non-message print line (1/2) . . . . . . . 7115. ISPF dialog defining the joblog monitoring of a

non-message print line (2/2) . . . . . . . 7116. Common MAT entry for message INGY1300I

and jobs defining a message id for monitoring . 7117. ISPF dialog defining the joblog monitoring of

all JESMSGLG messages (3/3) . . . . . . 7118. Common MAT entry for message INGY1300I

and jobs without defining a message id formonitoring . . . . . . . . . . . . . 72

19. Joblog monitoring status information . . . . 7320. Alert Communication Flow . . . . . . . 7621. Events in the Lifecycle of an Application 8322. SMF Processing with z/OS Offloader . . . . 8823. Stop Definitions for a Process . . . . . . 11324. Delete a File . . . . . . . . . . . . 11325. SSH Daemon Listening . . . . . . . . 11626. Example of a UNIX Message . . . . . . 11827. Sample Panel for Command Processing 15028. Sample Panel for Code Processing . . . . 15029. Focal Point Forwarding Definitions for

Systems . . . . . . . . . . . . . 15630. Fullscreen TAF Application Definition Panel 15731. Example Processing of a Primary WTOR 16832. SA z/OS Exit Sequence during SA z/OS

Initialization . . . . . . . . . . . . 170

33. Threshold Definitions for MVS ComponentLOGREC . . . . . . . . . . . . . 200

34. MESSAGES/USER DATA Policy Item forEntry/Type-Pair MVSESA/LOGREC. . . . 200

35. Threshold Definitions for MVS ComponentSMFDUMP . . . . . . . . . . . . 202

36. MESSAGES/USER DATA Policy Item forEntry/Type-Pair MVSESA/SMFDUMP . . . 203

37. Threshold Definitions for MVS ComponentSYSLOG . . . . . . . . . . . . . 204

38. MESSAGES/USER DATA Policy Item forEntry/Type-Pair MVSESA/SYSLOG . . . . 205

39. MVSDUMP Thresholds . . . . . . . . 20740. MVSESA AMRF Command Definitions 21041. JES2 DRAIN Specifications Panel . . . . . 21442. DISPACF Panel . . . . . . . . . . . 21543. DISPACF JES2 INITDRAIN Panel . . . . . 21644. JES2 SPOOLSHORT Recovery Definition 21745. DISPACF Command Response Panel 21846. Threshold Definitions for MVS Component

LOG . . . . . . . . . . . . . . 22547. MESSAGES/USER DATA Policy Item for

Entry/Type-Pair MVSESA/LOG . . . . . 22548. Automated Discovery Overview . . . . . 22849. Discovery Engine Overview. . . . . . . 23150. Preloader Processes . . . . . . . . . 23351. Flat File Processing . . . . . . . . . 23552. Example SDF Panels . . . . . . . . . 26653. Example SDF Tree Structure . . . . . . 26754. Status Descriptors Chained to Status

Components . . . . . . . . . . . . 26955. Example Tree Structure Definition. . . . . 27656. Example SDF Panel . . . . . . . . . 27857. Example Panel Definition Entry . . . . . 27858. The SUMMARY Report . . . . . . . . 30159. The XCFGROUP Report . . . . . . . . 30260. The ARM Group Report . . . . . . . . 30361. The ASDETAIL Report . . . . . . . . 30462. The EXCLUDE Report . . . . . . . . 30563. The CONSTRCT Report . . . . . . . . 30564. The CONSTRCT Report (second section) 30665. The CONSTRCT Report (third section) 30666. The SYMBOLS Report . . . . . . . . 30767. The SYMBOLS Report (second section) 30768. The KBIMPORT Report . . . . . . . . 30869. The KBMAP Report . . . . . . . . . 30970. The SUBSYS Report . . . . . . . . . 309

© Copyright IBM Corp. 1996, 2014 ix

|||

||||

|||

||

||

||||

x System Automation for z/OS: Customizing and Programming

Tables

1. System Automation for z/OS Library xix2. Application Start . . . . . . . . . . . 13. Application Stop (1) . . . . . . . . . . 24. Application Stop (2) . . . . . . . . . . 35. Observed Status Monitor Routines . . . . . 416. Health Status Return Codes . . . . . . . 457. Inform List Policy Items . . . . . . . . 778. Inform List Communication Methods . . . . 779. Code Processing Example for the INGALERT

Message ID . . . . . . . . . . . . 8010. Layout of the SMF Record . . . . . . . 8411. Format of INGPUSMF Utility Data Set Records 8612. SINGSAMP SA z/OS Sample Library Routines 9713. Example Customization Dialog Definitions for

sshd . . . . . . . . . . . . . . 11514. Pass description for sshd. . . . . . . . 11615. INGUSS Command and USS Pipe Summary 11716. Policy Entry Names and Types for Command

Receivers . . . . . . . . . . . . . 11917. Sample Names and Load Modules for a

TSO/Batch Environment . . . . . . . . 11918. Functions and Operators for SA and NetView

respectively . . . . . . . . . . . . 12119. Receiver Names and Related Policies 12720. RDS/SA Functions and NetView Operators 12821. WTOBUF Recovery Process . . . . . . . 13822. Auxiliary Storage Management . . . . . 14323. Example SYSTEM_SHUTDOWN Command

Processing Entry . . . . . . . . . . 162

24. Externalized Common Global Variables 24125. Global Variables to Enable Advanced

Automation (CGLOBALS) . . . . . . . 24226. Global Variables That Define the Installation

Defaults for Specific Commands . . . . . 25527. SDF Components . . . . . . . . . . 27328. Panel Definition Entry Description . . . . 27929. Input Columns for Address Space

Identification Mapping File . . . . . . . 29430. Output columns for Address Space

Identification Files . . . . . . . . . . 29431. Job Prefix Values . . . . . . . . . . 29432. Address Space Type Values . . . . . . . 29533. Input Columns for XCF Group Identification

Mapping File . . . . . . . . . . . 29534. Output Columns for XCF Group

Identification Mapping File . . . . . . . 29535. Input Columns for XCF Group Member

Identification Mapping File . . . . . . . 29636. Output Columns for XCF Group Member

Identification Mapping File . . . . . . . 29637. Input Columns for USS Process Identification

Mapping File . . . . . . . . . . . 29638. Output Columns for USS Process

Identification Mapping File . . . . . . . 29639. Input Columns for Policy Mapping File 29740. Output Columns for Policy Mapping File 29741. Input Columns for Variable Mapping File 29842. Output Columns for Variable Mapping File 299

© Copyright IBM Corp. 1996, 2014 xi

xii System Automation for z/OS: Customizing and Programming

Accessibility

Accessibility features help users with physical disabilities, such as restrictedmobility or limited vision, to use software products successfully. SystemAutomation for z/OS supports several user interfaces. Product functionality andaccessibility features vary according to the interface.

The major accessibility features in this product enable users in the following ways:v Use assistive technologies such as screen reader software and digital speech

synthesizer, to hear what is displayed on screen. Consult the productdocumentation of the assistive technology for details on using those technologieswith this product and screen magnifier software

v Operate specific or equivalent features using only the keyboardv Magnify what is displayed on screen.

The product documentation includes the following features to aid accessibility:v All documentation is available to both HTML and convertible PDF formats to

give the maximum opportunity for users to apply screen-reader softwarev All images in the documentation are provided with alternative text so that users

with vision impairments can understand the contents of the images.

Using assistive technologiesAssistive technology products, such as screen readers, function with the userinterfaces found in z/OS®. Consult the assistive technology documentation forspecific information when using such products to access z/OS interfaces.

Keyboard navigation of the user interfaceUsers can access z/OS user interfaces using TSO/E or ISPF. Refer to z/OS TSO/EPrimer, z/OS TSO/E User's Guide, and z/OS ISPF User's Guide Vol 1 for informationabout accessing TSO/E and ISPF interfaces. These guides describe how to useTSO/E and ISPF, including the use of keyboard shortcuts or function keys (PFkeys). Each guide includes the default settings for the PF keys and explains how tomodify their functions.

© Copyright IBM Corp. 1996, 2014 xiii

xiv System Automation for z/OS: Customizing and Programming

Dotted decimal syntax diagrams

Syntax diagrams are provided in dotted decimal format for users accessing theInformation Center using a screen reader. In dotted decimal format, each syntaxelement is written on a separate line. If two or more syntax elements are alwayspresent together (or always absent together), they can appear on the same line,because they can be considered as a single compound syntax element.

Each line starts with a dotted decimal number; for example, 3 or 3.1 or 3.1.1. Tohear these numbers correctly, make sure that your screen reader is set to read outpunctuation. All the syntax elements that have the same dotted decimal number(for example, all the syntax elements that have the number 3.1) are mutuallyexclusive alternatives. If you hear the lines 3.1 USERID and 3.1 SYSTEMID, youknow that your syntax can include either USERID or SYSTEMID, but not both.

The dotted decimal numbering level denotes the level of nesting. For example, if asyntax element with dotted decimal number 3 is followed by a series of syntaxelements with dotted decimal number 3.1, all the syntax elements numbered 3.1are subordinate to the syntax element numbered 3.

Certain words and symbols are used next to the dotted decimal numbers to addinformation about the syntax elements. Occasionally, these words and symbolsmight occur at the beginning of the element itself. For ease of identification, if theword or symbol is a part of the syntax element, it is preceded by the backslash (\)character. The * symbol can be used next to a dotted decimal number to indicatethat the syntax element repeats. For example, syntax element *FILE with dotteddecimal number 3 is given the format 3 \* FILE. Format 3* FILE indicates thatsyntax element FILE repeats. Format 3* \* FILE indicates that syntax element *FILE repeats.

Characters such as commas, which are used to separate a string of syntaxelements, are shown in the syntax just before the items they separate. Thesecharacters can appear on the same line as each item, or on a separate line with thesame dotted decimal number as the relevant items. The line can also show anothersymbol giving information about the syntax elements. For example, the lines 5.1*,5.1 LASTRUN, and 5.1 DELETE mean that if you use more than one of theLASTRUN and DELETE syntax elements, the elements must be separated by acomma. If no separator is given, assume that you use a blank to separate eachsyntax element.

If a syntax element is preceded by the % symbol, this indicates a reference that isdefined elsewhere. The string following the % symbol is the name of a syntaxfragment rather than a literal. For example, the line 2.1 %OP1 means that youshould refer to separate syntax fragment OP1.

The following words and symbols are used next to the dotted decimal numbers:v ? means an optional syntax element. A dotted decimal number followed by the ?

symbol indicates that all the syntax elements with a corresponding dotteddecimal number, and any subordinate syntax elements, are optional. If there isonly one syntax element with a dotted decimal number, the ? symbol isdisplayed on the same line as the syntax element, (for example 5? NOTIFY). Ifthere is more than one syntax element with a dotted decimal number, the ?symbol is displayed on a line by itself, followed by the syntax elements that are

© Copyright IBM Corp. 1996, 2014 xv

optional. For example, if you hear the lines 5 ?, 5 NOTIFY, and 5 UPDATE, youknow that syntax elements NOTIFY and UPDATE are optional; that is, you canchoose one or none of them. The ? symbol is equivalent to a bypass line in arailroad diagram.

v ! means a default syntax element. A dotted decimal number followed by the !symbol and a syntax element indicates that the syntax element is the defaultoption for all syntax elements that share the same dotted decimal number. Onlyone of the syntax elements that share the same dotted decimal number canspecify a ! symbol. For example, if you hear the lines 2? FILE, 2.1! (KEEP), and2.1 (DELETE), you know that (KEEP) is the default option for the FILE keyword.In this example, if you include the FILE keyword but do not specify an option,default option KEEP will be applied. A default option also applies to the nexthigher dotted decimal number. In this example, if the FILE keyword is omitted,default FILE(KEEP) is used. However, if you hear the lines 2? FILE, 2.1, 2.1.1!(KEEP), and 2.1.1 (DELETE), the default option KEEP only applies to the nexthigher dotted decimal number, 2.1 (which does not have an associatedkeyword), and does not apply to 2? FILE. Nothing is used if the keyword FILEis omitted.

v * means a syntax element that can be repeated 0 or more times. A dotteddecimal number followed by the * symbol indicates that this syntax element canbe used zero or more times; that is, it is optional and can be repeated. Forexample, if you hear the line 5.1* data area, you know that you can include onedata area, more than one data area, or no data area. If you hear the lines 3*, 3HOST, and 3 STATE, you know that you can include HOST, STATE, bothtogether, or nothing.

Notes:

1. If a dotted decimal number has an asterisk (*) next to it and there is only oneitem with that dotted decimal number, you can repeat that same item morethan once.

2. If a dotted decimal number has an asterisk next to it and several items havethat dotted decimal number, you can use more than one item from the list,but you cannot use the items more than once each. In the previous example,you could write HOST STATE, but you could not write HOST HOST.

3. The * symbol is equivalent to a loop-back line in a railroad syntax diagram.v + means a syntax element that must be included one or more times. A dotted

decimal number followed by the + symbol indicates that this syntax elementmust be included one or more times; that is, it must be included at least onceand can be repeated. For example, if you hear the line 6.1+ data area, you mustinclude at least one data area. If you hear the lines 2+, 2 HOST, and 2 STATE,you know that you must include HOST, STATE, or both. Similar to the * symbol,the + symbol can only repeat a particular item if it is the only item with thatdotted decimal number. The + symbol, like the * symbol, is equivalent to aloop-back line in a railroad syntax diagram.

xvi System Automation for z/OS: Customizing and Programming

How to send your comments to IBM

We appreciate your input on this publication. Feel free to send us any commentsyou might have.

If you have feedback to the manuals

If you have comments on the manuals, like clarity, accuracy, and completeness ofthe information, use the Feedback channel on IBM Knowledge Center to send yourcomments.1. Click Feedback > Email IBM Knowledge Center support at the bottom of IBM

Knowledge Center.2. Log in to the invoked mailbox; or if the Launch Application window is

displayed, choose one mailbox and log in. A new email is displayed after login.3. In the email body, write down your feedback. Please include the specific book

and topic name that you're commenting on.4. Send the email to the default recipient.5. SA z/OS team will respond to you by email as soon as possible.

If you have a technical problem

Use one of the following feedback methods:v Contact your IBM service representativev Call IBM technical supportv Visit the IBM zSeries support web page at www.ibm.com/systems/z/support/

When you send comments to IBM, you grant IBM a nonexclusive right to use ordistribute your comments in any way it believes appropriate without incurring anyobligation to you.

IBM or any other organizations will only use the personal information that yousupply to contact you about the issues that you submit.

© Copyright IBM Corp. 1996, 2014 xvii

http://www.ibm.com/support/knowledgecenter/SSWRCJ_3.5.0/com.ibm.safos.doc_3.5/kc_welcome-444.html

http://www.ibm.com/support/knowledgecenter/SSWRCJ_3.5.0/com.ibm.safos.doc_3.5/kc_welcome-444.html

http://www.ibm.com/systems/z/support/

xviii System Automation for z/OS: Customizing and Programming

About this publication

This publication describes how to adapt your completed standard installation ofIBM® Tivoli® System Automation for z/OS (SA z/OS) as described in IBM TivoliSystem Automation for z/OS Planning and Installation to your environment. Thispublication contains information on how to add new applications to automationand how to write your own automation procedures. It also contains informationabout how to add new messages for automated applications.

Who Should Use This PublicationThis publication is primarily intended for automation programmers responsible for:v Customizing system automation and the operations environmentv Developing automation procedures and other operations capabilities

PrerequisitesThroughout this book, it is expected that readers are familiar with the SystemAutomation for z/OS and the following documentation:v IBM Tivoli System Automation for z/OS Operator's Commands

v IBM Tivoli System Automation for z/OS Programmer's Reference

v IBM Tivoli System Automation for z/OS Defining Automation Policy

Where to Find More Information

The System Automation for z/OS LibraryTable 1 shows the information units in the System Automation for z/OS library:

Table 1. System Automation for z/OS Library

Title Order Number

IBM Tivoli System Automation for z/OS Planning and Installation SC34-2716

IBM Tivoli System Automation for z/OS Customizing and Programming SC34-2715

IBM Tivoli System Automation for z/OS Defining Automation Policy SC34-2717

IBM Tivoli System Automation for z/OS User's Guide SC34-2718

IBM Tivoli System Automation for z/OS Messages and Codes SC34-2719

IBM Tivoli System Automation for z/OS Operator's Commands SC34-2720

IBM Tivoli System Automation for z/OS Programmer's Reference SC34-2748

IBM Tivoli System Automation for z/OS Product Automation Programmer'sReference and Operator's Guide

SC34-2714

IBM Tivoli System Automation for z/OS TWS Automation Programmer's andOperator's Reference Guide

SC34-2749

IBM Tivoli System Automation for z/OS End-to-End Automation Adapter SC34-2750

IBM Tivoli System Automation for z/OS Monitoring Agent Configuration andUser's Guide

SC34-2751

SA z/OS Home Page:

© Copyright IBM Corp. 1996, 2014 xix

For the latest news on SA z/OS, visit the SA z/OS home page at:http://www-03.ibm.com/software/products/en/tivosystautoforzos

Related Product InformationFor information that supports Tivoli System Automation for z/OS, visit the z/OSlibrary in IBM Knowledge Center (https://www.ibm.com/support/knowledgecenter/SSLTBW).

Summary of Changes for SC34-2715-00This document contains information previously presented in System Automationfor z/OS V3R4.0 Customizing and Programming, SC34-2644-00.

You may notice changes in the style and structure of some content in thisdocument—for example, headings that use uppercase for the first letter of initialwords only, and procedures that have a different look and format. The changes areongoing improvements to the consistency and retrievability of information in ourdocuments.

This document contains terminology, maintenance, and editorial changes.

New InformationRead-only Variables

AOFSYSPLEX, and AOFSYSPLEXGROUP are added in “Read-OnlyVariables” on page 241.

Global Read/Write VariablesAOF_AAO_AOCUPDT_LOGGING, AOF_AAO_HW_VALIDATION,AOF_AAO_IPL_COMPLETE_MSG, AOF_AAO_SDF_MSGDEL_OPERID,AOF_AAO_SDFROOT, AOF_AAO_SEC_PPIAUTH, AOF_AAO_SDFCxxx.*AOF_AAO_SDFCSAPLEX, AOF_AAO_SHUTDOWN_STOPAPPL, andINGPAC_SHOWNOLIMIT are added in “Read/Write Variables” on page242.

AOFEXC25AOFEXC25 is added to the list of Command Exits.

INGRYSHUThe INGRYSHU automation routine is added in Automation Solutions.

Parameter Defaults for CommandsNew default parameters for panel column layout are supplied for thecommands DISPGW, DISTMTR, DISPSTAT, INGAMS, INGIMS andINGLIST in “Parameter Defaults for Commands” on page 255. The variableINGPAC_WAIT is also added.

Guidelines for using RDS for TSOA new section is added to provide guidelines of how to customizeRelational Data Services under TSO including setting up security profiles.See “Use RDS for TSO” on page 128.

Command Information RetrievalA REXX program to retrieve originating information when using theAOFRYCMD command is described in “Command Information Retrieval”on page 123.

xx System Automation for z/OS: Customizing and Programming

|||

http://www-03.ibm.com/software/products/en/tivosystautoforzos

https://www.ibm.com/support/knowledgecenter/SSLTBW

https://www.ibm.com/support/knowledgecenter/SSLTBW

Looping Address Space SuppressionThe capacity to suppress looping address spaces is defined in “LoopingAddress Space Suppression” on page 192 of Chapter 18, “AutomationSolutions,” on page 189.

Changed InformationJob Log Monitoring

This chapter is updated to include examples for a multi-step job. SeeChapter 5, “Joblog Monitoring,” on page 67.

Relational Data TablesThe capacity to “Enable/Disable Persistent Relational Data Tables” on page127 is now described as part of Relational Data Services in Chapter 11,“Enabling Relational Data Services (RDS),” on page 127.

Parameter Defaults for CommandsThe parameter defaults are updated for the INGRUN command. See“Parameter Defaults for Commands” on page 255.

Add SA TSO REXX LibraryThis former section is retitled to “Customize the TSO Environment” onpage 130 with new details for this task.

Moved InformationRACF Protection of INGRCRDX under TSO

This section is moved to IBM Tivoli System Automation for z/OS Planning andInstallation to be part of installing function packages for NetView and TSOin the chapter "Security and Authorization". This section included "Enableand Disable RACF Checking" and "RACF Profile and Class". "Install SAProvided Authorized Command" is part of the step "Configure FunctionPackages for TSO" in "Traditional SA z/OS Configuration".

Deleted InformationAOFEXC12

This command exit is removed.

Defining an Outbound Gateway AutotaskThis task is removed from "Defining Gateway Sessions" in Chapter 13,“Automating Networks,” on page 155.

About this publication xxi

xxii System Automation for z/OS: Customizing and Programming

Chapter 1. Adding a New Application to Automation

This information outlines the requirements to add and monitor a new applicationfor SA z/OS.

Preparation Before Automating an ApplicationBefore you can automate a product you need to extract its characteristics like itsstart and stop behaviour and parameters like its jobname.

The following steps should help you to obtain these characteristics. Once you havefinished you need to add the application to your automation policy. Refer to IBMTivoli System Automation for z/OS Defining Automation Policy for this activity. Themain requirements for the automation of an application are:v Address Space propertiesv Application Startv Application Shutdownv Application Eventsv Application Monitoringv Outstanding Reply Processingv Topology

Address Space propertiesWhen adding an new application, you need to know the following most importantcharacteristics of the application:v Jobnamev JCL procedure namev Is it scheduled by the Master Scheduler or a scheduling subsystem?v Is it an MVS™, OMVS application or another kind of application (for example a

NetView task)?v Location for running the application: every sysplex's system, once in the sysplex

or on a subset of systems within the sysplex?

The application's general properties are mostly defined in the APPLICATIONINFO policy.

Step 1 - Application StartBefore you can introduce a new application you should consider how it is startedand all the actions required to make it operational.

Therefore it is important to know:

Table 2. Application Start

Actions Required Available Functionality

Are there any actions to complete before theapplication itself could be started?

To include these actions in automation, usethe application's PRESTART policy. Anycommand specified there is issued prior tothe insertion of the start command.

© Copyright IBM Corp. 1996, 2014 1

Table 2. Application Start (continued)


What is the application's start command?And are there different start commandsdepending on the startup mode of anapplication (for example, the normal and thelight start for a DB2® database)?

The start command should be located in theSTARTUP policy. It also provides the fullflexibility for different start commands byspecifying different start types. Once a starttype is set, the specified command will bechosen. The start type easily can bechosen/changed at System Automationruntime.

Who starts the application when it is notstarted by System Automation? Is it startedby another instance?

The APPLICATION INFO policy lets youspecify an EXTERNAL STARTUP parameter.

Are there any actions to complete after theapplication initialization?

Use the POSTSTART policy to issueadditional commands after the fullinitialization of the application.

Note: All startup policies support flexible start types.

Step 2 - Application StopOnce the application is no longer required, you need to take all the necessary stepsto bring it down in a planned way.

Therefore it is important to know:

Table 3. Application Stop (1)


Should you issue commands to prepare theapplication shutdown ?

Use the SHUTDOWN INIT policy to identifyadditional commands to be issued before theapplication termination can be initiated.

Which command initiates the terminationprocess? And what happens when the stopcommand does not take effect?

System Automation has the concept ofcommand escalation. It provides the capacityto specify an order of terminationcommands. System Automation will issuethe first command and verify the effectbefore it inserts the next more effectivecommand. There are three policies(SHUTDOWN NORM, IMMED, FORCE)where you can specify different shutdowncommand sequences for different shutdowntypes.

Who stops the application when it is notstopped by System Automation? Is thereanother instance controlling theapplication?

The APPLICATION INFO policy lets youspecify an EXTERNAL SHUTDOWNparameter.

Are there any final termination actions tocomplete after an orderly applicationtermination?

Use the SHUTDOWN FINAL policy to issueadditional commands after the terminationof the application.

Sometimes it can happen that an application terminates unexpectedly. In this caseit might be necessary to complete some cleanup actions before the application canbe restarted. Consequently it is necessary to know:

How to Add a New Application to Automation

2 System Automation for z/OS: Customizing and Programming

Table 4. Application Stop (2)


Are there any necessary cleanup activitiesto be completed before the application canbe restarted?

The concept of status commands addressesthis issue. Once the application reaches aspecific status, the defined command will beissued.

Is the application restartable in case of anunexpected termination?

System Automation recognizes severaltermination situations for applications.Depending on the situation SystemAutomation is able to distinguish between arecoverable and an unrecoverable error. As aresult, System Automation determineswhether to restart the application or not.This concept is Code Match processing.Additionally the RESTART option in theAPPLICATION INFO policy defines thecircumstances when System Automationshould restart the application.

Is the application restarted automaticallyby another application? Is the applicationARM (Automatic Restart Manager) enabledand will it be restarted automatically?

System Automation provides the concept ofMove groups to accomplish the samebehavior as the ARM mechanism does. It isrecommended to use Move groups forachieving high availability of applications.

Step 3 - Application EventsSystem Automation reacts to events. More specifically, it reacts to messages sent byapplications or the system itself.

There are many kinds of applications. Each of them sends a varying degree ofmessages which can be used to determine its status. The messages representdifferent states during an application's life-cycle. Normally MVS resources provideproper messages to determine the status of an application. Resources within OMVSare mostly silent.

Step 3 points you to important messages in the life-cycle of the application. As youcan see below a resource is started once. After an amount of initialization time it isfully operational. When the resource is no longer needed a stop command isinvoked to terminate it. After the termination processing it does not exist anylonger.


Chapter 1. Adding a New Application to Automation 3

What are the important messages at the points in the application's life-cycle asillustrated in Figure 1 above?

For example a message IEF403I is sent when the system observes that anapplication has been started. IEF404I is issued when the application terminated.

Are there are other kinds of usable events at the specific points?

Step 3 discusses also the situation of an unplanned application termination.

Are there important messages at point 3 and 4 of Figure 1 in case of an unplannedtermination?

For example a message IEF450I points to an unplanned termination.

Step 4 - Application MonitoringAutomating resources does not only consist of starting and stopping the resources.It is also important to know methods that determine whether the component isworking as expected or already terminated. System Automation provides propermonitoring routines to be able to determine its state.

What kind of an application is it and could the state of the application bedetermined by:v The existence of an address spacev The existence of a USS processv The status of a NetView task

The corresponding monitor could be specified in the field MONITOR ROUTINE.

Step 4 presents messages issued by applications and how communicative they are.So it is useful to decide whether an application must be actively monitored or itsstate could be reliably derived from messages. If this is the case the MONITORINTERVAL could be set to NONE. It helps to reduce the messages in the NetViewlog and to reduce unnecessary system activity. The monitoring action itself takesplace at the startup and shutdown cycle to verify the state of the application.

Termination Complete

Initialization Complete Stop Command

3

Start Command

1

2

4

Figure 1. Application Lifecycle



In contrast, there are less communicative applications. In this case a well balancedmonitor interval (specific enough) ensures a periodic monitoring service to verifythe applications status.

Sometimes it is not enough to know whether a resource is running or not. Manysituations require more detailed information as well as its status. The concept ofmonitor resources provides the infrastructure to evaluate the status of resources indetail and to react properly to the specific situation.

Outstanding Reply ProcessingSA z/OS keeps track of all outstanding Write-to-Operator Replies (WTORs) that itreceives if it does not reply to them immediately. Because some applications mayhave more than one outstanding WTOR at the same time, and not all WTORs areequally important, they are classified accordingly.

For more details refer to Chapter 16, “WTOR Processing,” on page 165.

TopologyNormally the application to be automated depends on the underlyinginfrastructure, like JES2 or TCPIP. This means that this infrastructure must beavailable before you can start the application.

Vice versa the application can be a prerequisite for other applications, before theycan be started.

Likewise you need to think about which other applications must be terminatedprior to the termination of the application.

As described above, there are relationships between the applications.

At this point it might be helpful to draw a picture and to visualize therelationships between the application in case of a start and a stop situation.

SA z/OS provides Best Practice policies containing solutions for several products.The solutions are illustrated in PDF file format located in: /usr/lpp/ing/doc/policies.

Please refer to the appropriate file to find out more information about the solutionyou are trying to automate.

Adding the Application to Automation

Define an Application Policy ObjectTo add a new application to SA z/OS, you must create and define a newApplication policy object using the SA z/OS customization dialog.

With the customization dialog, you also define how the new application should beautomated by SA z/OS, for example:v Specifying startup or shutdown commands for the application,v Specifying the appropriate monitoring routine,v Specifying relationships to correlate it with other applications,v Linking the application into an application group,v Considering where the applications should be visible.


Chapter 1. Adding a New Application to Automation 5

SA z/OS provides Best Practice policies containing solutions for several products.

How to add a new application and how to access System Automation's BestPractice solutions is described in detail in IBM Tivoli System Automation for z/OSDefining Automation Policy.

Using Automated Discovery to Define Application PolicyObjects

This functionality is primarily intended to build a basic automation policy. You candiscover z/OS system resources like address spaces and along with SA z/OSdelivered policies you can map the discovered address spaces to applicationsdefined in the SA z/OS delivered policies.

Thus, a new user policy can be populated with APL policy objects.

For more information, refer to Chapter 19, “Automated System ResourceDiscovery,” on page 227.

Build New Automation Configuration FilesWhen you finish defining the application in the customization dialog, build thenew automation configuration files from the updated policy database.

See IBM Tivoli System Automation for z/OS Defining Automation Policy for moreinformation.

After you have completed this step, the application is known to SA z/OS and cantherefore be automated according to the policy that was defined in “Define anApplication Policy Object” on page 5.



Chapter 2. Creating Automation Procedures

You can write additional automation procedures to supplement the basicautomation procedures that are supplied by SA z/OS. For example, you may wantto develop procedures to automate an application that is used exclusively on yoursystem or to perform specialized automated operations for a subsystem.

SA z/OS commands and routines perform basic functions such as loggingmessages and checking automation flags. You can use them in your ownautomation procedures.

SA z/OS automation routines are convenience routines that provide yourautomation procedures with a simple, standard way of interfacing with theautomation control file, automation status file, and NetView log file. It is stronglyrecommended that you use these routines wherever possible in your own code.

“How Automation Procedures Are Structured” on page 8 describes how tostructure your automation procedures. Refer to IBM Tivoli System Automation forz/OS Programmer's Reference for detailed descriptions and examples of theautomation routines and file manager commands you can use in your automationroutines.

How Automation Procedures Are CalledThere are several ways to call an automation procedure including:v Calling the automation procedure from the NetView automation table using

SA z/OS automationv Keying in the automation procedure name or its synonym into a NetView

command linev Calling the automation procedure from another programv Starting the automation procedure with a timerv Starting the automation procedure with the NetView EXCMD commandv Starting the automation procedure on an automation operator with the SA z/OS

AOFEXCMD command routinev In the customization dialog, entering your automation procedure name in the

Command text or Command field of various policy items for the followingentry types:– Application– MVS Component– Timers– Monitor Resources

Note: Not all routines can be called through all interfaces as some requireextensive environmental setup before they are invoked.


How Automation Procedures Are StructuredIt is recommended that the structure of automation procedures contain three mainparts, as follows:1. Perform initialization processing2. Determine whether automation is allowed3. Perform automation processing.

Figure 2 illustrates the structure of automation procedures for system operationsand Figure 3 for processor operations.

Figure 2. Automation Procedures for System Operations

Figure 3. Automation Procedures for Processor Operations

How Automation Procedures Are Structured


The following sections provide more details about each part of an automationprocedure.

Performing Initialization ProcessingInitialization processing may not be required for simple automation procedures.

Initialization processing is responsible for:v Setting up any error trap routines.v Identifying the automation procedure by setting a local variable either explicitly

or at execution time. This step makes it simpler to code routines that logmessages and send notifications.

v Declaring the global variables, such as common and task global variables, thatare used for subsystem definition values in the automation procedure.See “Global Variables” on page 199 for descriptions of global variables.

v Checking whether debugging is on.v Issuing debugging messages, if debugging is turned on.v Validating the automation procedure call.

This step can help prevent an operator from calling the automation procedureinappropriately. Automation procedures can also be validated using commandauthorization checking methods provided by NetView or an SAF product.

v Saving NetView message parameters. This step is necessary if your automationprocedure uses the NetView WAIT statement and you need to access the originalmessage text or control information.

For more information on coding automation procedure initialization sections, referto “Example Automation Procedure” on page 16, to Tivoli NetView for z/OSCustomization Guide and to Tivoli NetView for z/OS Automation Guide.

Determining whether Automation Is Allowed

System OperationsAutomation procedures for applications and MVS components that are called fromthe NetView automation table should always perform an automation check bycalling the AOCQRY automation routine. AOCQRY checks that the automationflags allow automation.

These checks eliminate the risk of automating messages for applications thatshould not be automated, or for which automation is turned off. AOCQRY alsoinitializes most of the common and task global variables that are used in theautomation procedure with values specific to the application.

Refer to IBM Tivoli System Automation for z/OS Programmer's Reference for moreinformation on coding the automation check routine.

Processor OperationsMost of the processor operations commands run only when processor operationshas been started. To determine whether processor operations is active, you can usethe ISQCHK command in your automation routines.

If processor operations is not running, ISQCHK returns return code 32 and issuesthe message:ISQ0301 Cannot run cmd-name command until Processor Operations has started.


Chapter 2. Creating Automation Procedures 9

Your application can then issue the ISQSTART command to begin processoroperations.

Performing Automation ProcessingAutomation processing is performed by any combination of SA z/OS routines andyour own code.

The following documentation gives more information on coding automationprocedures:v “Automation Processing in System Operations”v “Automation Processing in Processor Operations” on page 12

Automation Processing in System OperationsThis section contains information on how to customize automation processing forsystem operations.

Updating Status Information:

You can update status information by calling AOCUPDT. This routine is usedwhen a message indicates a status change.

This would normally be done from the automation routines ACTIVMSG,HALTMSG, and TERMMSG. Making your own status updates may causeunpredictable results.

For more information, see IBM Tivoli System Automation for z/OS Programmer'sReference.

Logging Messages and Sending Notifications:You can log messages and send notifications by calling AOCMSG.

AOCMSG performs the following actions:v Formats a message for display or loggingv Issues messages as SA z/OS notification messages to notification operators


Issuing Commands and Replies:

You can issue commands and replies by calling ACFCMD and ACFREP.

You can use these routines to:v Issue one or more commands in response to a message.v Issue a single reply in response to a message.v Use the step-by-step (PASS) concept to react to or recover from an automation

event.

ACFCMD issues one or more commands. It supports both a single reaction and thestep-by-step (PASS) concept. For more information, see IBM Tivoli SystemAutomation for z/OS Programmer's Reference.



ACFREP issues a single reply. It supports both a single reaction and thestep-by-step (PASS) concept. For more information see IBM Tivoli SystemAutomation for z/OS Programmer's Reference.

In many cases you may be able to use the ISSUEACT automation procedure thatalso supports single and pass processing.

Checking Thresholds:

You can check and update thresholds by calling the CHKTHRES automationroutine. Use CHKTHRES to track and maintain a threshold, and to change therecovery action based on the threshold level exceeded.

For more information see IBM Tivoli System Automation for z/OS Programmer'sReference.

Checking Error Codes:

You can check error codes by calling CDEMATCH. It compares error codes in amessage to a set of automation-unique error codes to determine the action to take.


In some cases you may be able to use the code matching capabilities of ISSUEACTand TERMMSG.

Using File Manager Commands:

You can use file manager commands to access SA z/OS control files such as theautomation control file and automation status file:v Use the ACF command if you need to load or display the automation control

file.v Use the ACFFQRY command to query the automation control file quickly.v Use the ASF command to display the automation status file.v Use the ASFUSER command to modify the automation status file fields reserved

for your own information.


Using External Code for Timers, Logic, and Other Functions:

Your automation procedures may require code to set timers, to perform logicunique to your enterprise or to the automation procedure itself, and to performother functions.

Some examples include:v Issuing commands and trapping responses.

You can issue commands and trap responses using the NetView WAIT or PIPEcommands. You may need to use these commands in your code if it is necessaryto check the value or status of a system component or application beforecontinuing processing. For more information, see Tivoli NetView for z/OSCustomization Guide.

v Setting Common Global and Task Global values to control processing.



You can set Common and Task Global values by using NetView commands. Youmay need to set these values if it is necessary to set a flag indicating progress,message counts, and other indicators that must be kept from one occurrence of amessage to the next. See IBM Tivoli System Automation for z/OS DefiningAutomation Policy for a table of all externalized SA z/OS global variables.Also refer to the discussion of common and task global variables in TivoliNetView for z/OS Customization Guide.

v Setting timer delays to resume processing.You can set timer delays by using the NetView AT, AFTER, EVERY and CHRONcommands. You can use these commands when an automation procedure musteither resume processing or initiate another automation procedure after a giventime to do additional processing. For example, you could use these commandsto perform active monitoring of subsystems. For more information, see thediscussion of AT, AFTER, EVERY and CHRON commands in Tivoli NetView forz/OS Automated Operations Network User's Guide.

Automation Processing in Processor OperationsThis section contains information on how to customize automation processing forprocessor operations.

Initializing a Target System:

If your routines need to start target systems (hardware and/or operating system),issue the ISQCCMD ACTIVATE command.

Shutting Down a Target System:

If your routines need to shut down a target system, issue the ISQCCMDDEACTIVATE OCF command.

Before issuing the command to close the target system, shut down all of yourfunctioning subsystems. This avoids any unexpected situations at the targetsystem.

Issuing Other OCF Commands:

All OCF commands supported by processor operations can be issued fromautomation routines.

See IBM Tivoli System Automation for z/OS Operator's Commands for details aboutthese commands.

Reserved SA z/OS Commands:

The SA z/OS commands ISQISUP, ISQISTAT, ISQCMMT, ISQSTRT, ISQXIPM,ISQGPOLL, and ISQGSMSG are not intended for your use. Do not use these inyour automation routines.

Unexpected results may occur.

The following commands can only be used from an operator console and shouldnot be used in your automation routines or with ISQEXEC: ISQXDST, ISQXOPT,and ISQHELP.

The following commands are for automation and should not be used in yourautomation routines: ISQI101, ISQI212, ISQMCLR, ISQI320, ISQIUNX, ISQI347,



ISQI470, ISQI886, ISQI888, ISQI889, ISQI128, ISQIVMT, ISQMVMI1, ISQMVMI2,ISQMWAIT, ISQMDCCF, ISQM020, and ISQIPLC.

Serializing Command Processing:

Serializing command processing ensures that commands and automation routinesare processed in the order in which they are sent to a target system console. It canalso prevent the command sequence from being interrupted by other tasks.

Specific target control tasks are assigned to specific target systems duringinitialization of the target system. More than one target system can share a targetcontrol task, but a target system never has more than one target control taskallocated to it to perform work.

When a command or an automation routine is sent to a target system, it can beprocessed partly in the issuing task (a logged-on operator or an autotask) andpartially in a target control task. When the command or automation routine is tobe processed by a target control task, it is either allocated to the target control taskand processed, or queued to be processed by the target control task. This serializesthe processing of commands and automation routines. Serializing ensures that theyare processed in the order in which they were sent to the target system console.

The NetView program has priority defaults established during its initialization.Usually, everything running under NetView has a low priority. You can use theNetView DEFAULTS command to see what the settings are, but you should notchange them. For SA z/OS command processing to be serialized as designed, allcommands used in SA z/OS must have a priority setting of “low”. If you changethe priorities or have more than one priority for commands used in SA z/OS, thedifference in the priorities may defeat the serialization that results from thearchitecture of the target control task.

Sending an Automation Routine to a Target Control Task:

If you run the same series of SA z/OS commands regularly, you can program thecommands into a NetView automation routine. Follow the guidelines you use forany NetView automation routine.

A NetView autotask or a logged-on operator can then run this routine or send it toa target control task. Use the following command to transfer an automation routineto a target control task:ISQEXEC target-system-name SC routine-name

When you issue the ISQEXEC command to process an automation procedure, all ofthe commands are processed in the order in which they occur in the automationprocedure. This is because the ISQEXEC command sends work to a target controltask, which processes commands serially. Any other commands or automationroutines issued to the same console by the ISQEXEC command are queued forprocessing by the target control task and do not start until the previous commandor automation procedure completes.

The ISQEXEC command also frees the original task from any long-runningcommand sequence. This lets you use the issuing task, such as an OST, for otherwork.

The ISQEXEC command does not lock consoles to ensure command serialization;the command serialization process is due to the target control task allocation



scheme. Commands and automation routines are processed in the order in whichthey occur; however, it is possible for commands from other tasks to interrupt thecommand sequence.

For more information about the ISQEXEC command, see IBM Tivoli SystemAutomation for z/OS Operator's Commands.

Locking a Console:

Several routines and operators may attempt to address the same console at thesame time. The ISQEXEC command does not prevent other tasks from interruptingthe sequence of commands being processed by the target control task; it does notlock the console.

To prevent a sequence of commands from being interrupted, use the ISQXLOC andISQXUNL commands. The ISQXLOC command locks access to the console. If atask attempts to issue a command to a locked console, the task is told that theconsole is locked, and the command fails. When you are finished with thesequence of commands that must be processed without interruption, issue theISQXUNL command to unlock access to the console.

You can use the ISQXLOC and ISQXUNL commands within automation routines toensure that they complete without interference from other tasks. For automationroutines that issue a number of SA z/OS commands, put the following commandafter the ISQEXEC command and near the beginning of the routine:ISQXLOC target-system-name SC

This locks access to the target system console to the current task until the lock isdropped by the command:ISQXUNL target-system-name SC

Only the task that issued ISQXLOC can successfully issue ISQXUNL. If anISQXLOC command is issued from a locked sequence of commands, it is rejectedbecause the console is already locked.

When you lock a system console for a target system running on a logical partition,you lock that system console for all other target systems using that processor. Acommand sent to a system console for any other target system (logical partition)on that target hardware definition does not run until the console is unlocked.

If your automation routine cannot wait for a console to be released, use theISQOVRD command to gain control of the console. Use the following commandonly in critical automation routines:ISQOVRD target-system-name SC

When the routine issuing the override command completes, the lock is removedand the console is available.

How to Make Your Automation Procedures GenericBy using the SA z/OS automation routines, you can make your own automationprocedures generic.

A generic automation procedure comprises three parts. For each part, there arespecial automation routines that help you to fulfill your tasks:



PreparationCheck if automation is allowed and should be done. Use automationroutine AOCQRY.

EvaluationWhat should be done? Use automation routine CDEMATCH.

ExecutionDo what should be done. Use automation routine ACFCMD or ACFREP.

For more information on the mentioned automation routines refer to IBM TivoliSystem Automation for z/OS Programmer's Reference. For more information oncommand processing or reply processing refer to IBM Tivoli System Automation forz/OS User's Guide.

Processor Operations CommandsWhenever possible, your automation routines should make use of SA z/OSprocessor operations OCF commands, also called common commands. Thesecommands are independent of the hardware type of the target system's processor.Therefore, the use of these commands minimizes the need for changes to yourautomation routines if you need to add new processors to your configuration. SeeIBM Tivoli System Automation for z/OS Operator's Commands for a detaileddescription of the processor operations commands.

Developing Messages for Your Automation ProceduresDepending on the scope of additional programming, creating new automationprocedures may also require developing additional messages.

Some SA z/OS facilities and commands you can use to develop messages include:

*************************************************** Preparation ***************************************************

AOCQRY

- check if the resource is controlled by SA z/OS

- check if automation is allowed

- prepare/set task global variables for CDEMATCH, ACFCMD and ACFREP

...

CDEMATCH

- code matching (table search in ACF)

- find out required action

...

ACFCMD/ACFREP

- do required action:issue command / respond reply

Figure 4. Skeleton of an Automation Procedure

How to Make Your Automation Procedures Generic


v The AOCMSG automation routine (see IBM Tivoli System Automation for z/OSProgrammer's Reference).

v The AOCUPDT automation routine (see IBM Tivoli System Automation for z/OSProgrammer's Reference).

The following steps summarize the message development process.1. Choose a message ID. Make sure it is unique.2. Use NetView message services to define the message to NetView.

Put an entry for the message in a DSIMSG data set. This data set must beidentified in a DSIMSG data definition (DD) name.

3. Use the AOCMSG automation routine to issue the message (see IBM TivoliSystem Automation for z/OS Programmer's Reference).

4. Add an entry for the message to your production copy of the NetViewDSIMSG data set.

Example AOCMSG CallThis example shows how to code AOCMSG to issue message ABC123I.

Entries for messages in DSIMSG member DSIABC12 are as follows:*********************************************************120I ...121I ...122I ...123I 10 40 THE EAGLE HAS &1124I ...*********************************************************

Your automation procedure contains the following AOCMSG call:<other automation procedure code>...

AOCMSG LANDED,ABC123...<other automation procedure code>

When AOCMSG is called as specified in the automation procedure, DSIMSGmember DSIABC12 is searched for message ABC123I. Substitution for variable &1occurs, and the following message is generated:ABC123I THE EAGLE HAS LANDED

Note that the message is defined with a 10 and a 40 between the message ID andthe first word of the message. These are the SA z/OS message classes to which themessage belongs. When the message is issued a copy is sent to every notificationoperator who is assigned class 10 or class 40 messages.

Refer to Tivoli NetView for z/OS Customization Guide for further information ondeveloping new messages.

Example Automation ProcedureThis section provides an example of an application program that handles a z/OSmessage. The automation procedure uses a subset of the SA z/OS automationroutines./* Example SA z/OS Automation Procedure */

▌1▐ Signal on Halt Name Aof_Error; Signal on Failure Name Aof_Error

Developing Messages for Your Automation Procedures


Signal on Novalue Name Aof_Error; Signal on Syntax Name Aof_Error

▌2▐ Parse source . . ident .

▌3▐ "GLOBALV GETC AOFDEBUG AOF."||ident||".0DEBUG AOF."||ident||".0TRACE"If AOFDEBUG = ’Y’ Then"AOCMSG "||ident||",700,LOG,"||time()||","||opid()||","||Arg(1)

loc.0debug = AOF.ident.0DEBUGloc.0trace = AOF.ident.0TRACEloc.0me = identIf loc.0trace <> ’’ Then Doloc.0debug = ’’Trace Value loc.0trace

End

▌4▐ save_msg = msgid()save_text = msgstr()lrc = 0

▌5▐ /* This procedure can only be called for msg IEA099A */If save_msg <> ’IEA099A’ Then Do"AOCMSG "||loc.0me||",203,"||time()||","||opid()Exit

End

▌6▐ "GLOBALV GETC AOFSYSTEM"cmd = ’AOCQRY ’||save_msg||’ RECOVERY ’||AOFSYSTEMcmdsvretcode = rcIf loc.0debug = ’Y’ Then"PIPE LIT /Called AOCQRY; Return Code was "||svretcode||"/" ,

"| LOGTO NETLOG"

/* -------------------------------------------------------------- **** Check return code from AOCQRY **** 0 = ok 1 = global flag off **** 2 = specific flag off 3 = resource not in ACF **** 4 = bad parms 5 = errors/timeout **** -------------------------------------------------------------- */Select

▌7▐ When svretcode >= 3 Then Do"AOCMSG "loc.0me",206,,"time()",,,"cmd",RETCODE="svretcodelrc = 1

End▌8▐ When svretcode > 0 Then Do

"GLOBALV GETT AUTOTYPE SUBSAPPL SUBSTYPE SUBSJOB""AOCMSG "loc.0me",580,,"time()","SUBSAPPL","SUBSTYPE"," ,SUBSJOB","AUTOTYPE","save_msg

lrc = 1EndOtherwise Do

▌9▐ Parse Var save_text With . ’JOBNAME=’ save_job ’ASID=’ save_asid .

▌10▐ ehkvar1 = save_jobehkvar2 = save_asid"GLOBALV PUTT EHKVAR1 EHKVAR2"

▌11▐ cmd = ’ACFCMD ENTRY=’||AOFSYSTEM||’,MSGTYP=’||save_msgcmdsvretcode = rcIf loc.0debug = ’Y’ Then"PIPE LIT /Called ACFCMD; Return Code was "||svretcode||"/" ,

"| LOGTO NETLOG"

/* ---------------------------------------------------------- **** Check return code from ACFCMD **** 0 = ok 1 = no commands found in ACF **** 4 = bad parms 5 = errors/timeout **** ---------------------------------------------------------- */

Example Automation Procedure


▌12▐ If svretcode > 1 Then Do"AOCMSG "loc.0me",206,,"time()",,,’"cmd"’,RETCODE="svretcodelrc = 1

EndEnd

End /* End of Select svretcode */

▌13▐ Exit lrc

▌14▐ Aof_Error:Signal Off Halt; Signal Off FailureSignal Off Novalue; Signal Off Syntaxerrtype = condition(’C’)errdesc = condition(’D’)SelectWhen errtype = ’NOVALUE’ Then rc = ’N/A’When errtype = ’SYNTAX’ Then errdesc = errortext(rc)Otherwise Nop

End"AOCMSG "errtype",760,,"loc.0me","sigl","rc","errdescExit -5

Notes on the Automation Procedure Example▌1▐ This step sets error traps for negative return codes, operator halt

commands, and REXX programming errors.

▌2▐ This step defines the identity of the automation procedure.

▌3▐ This step handles the debug and trace settings (refer to “UsingAOCTRACE to Trace Automation Procedure Processing” on page 20.

▌4▐ Save the NetView message variables the automation procedure uses.

▌5▐ Perform authorization check. This procedure can only be called for aparticular message.

▌6▐ This section performs the automation check:1. Fetch the AOFSYSTEM common global variable that contains the

information under which entry name the system messages are stored inthe automation control file (ACF).

2. The automation procedure calls the AOCQRY command. This performsthe automation flag check and presets some task global variables thatare used by other automation routines like ACFCMD.

▌7▐ Issue message AOF206I if call to AOCQRY fails.

▌8▐ Issue message AOF580I if automation flag is off.

▌9▐ Get the job name and asid reported in the message.

▌10▐ Set EHKVARn variables for ACFCMD.

▌11▐ Call ACFCMD to issue the command specified in the configuration files.The Automation Control File entry for the message IEA099A could looklike this:

MVSESA IEA099A,CMD=(,,’MVS C &EHKVAR1,A=&EHKVAR2’)

▌12▐ Issue message AOF206I if call to ACFCMD fails.

▌13▐ Exit with return code that indicates successful or unsuccessful processing.

▌14▐ This code logs a message if an error is trapped at step ▌1▐.

Example Automation Procedure


Installing Your Automation ProceduresThe installation process for a new automation procedure depends on the languagein which the automation procedure is written.v If the automation procedure uses a compiled language, such as PL/I, C, or

Assembler:1. Compile or assemble your source into an object module.2. Link-edit the object module into a NetView load library.3. Include an entry for the automation procedure in the CNMCMDU member

of the NetView DSIPARM data set.v If the automation procedure uses an interpreted language such as NetView

command list or REXX:1. Copy the automation procedure into a NetView command list library2. Optionally include an entry for this automation procedure in the

CNMCMDU member of the NetView DSIPARM data set. Then it is morequickly found and invoked.

For more information on preparing your code for use and installing it, refer toTivoli NetView for z/OS Customization Guide.

Testing and Debugging Automation ProceduresThis section describes SA z/OS and NetView facilities you can use for testingautomation procedures, including:v SA z/OS assist modev SA z/OS AOCTRACE operator facilityv NetView testing and debugging facilities

The Assist Mode FacilitySA z/OS provides an assist mode facility, so that you can verify automated actions.

When assist mode is on, actions that are normally taken by SA z/OS automationprocedures, such as issuing a command or reply, are not performed. Insteadmessages that describe what would have happened are written to the netlog.

The assist mode is associated with automation flags (Automation, Initstart, Start,Recovery, Terminate or Restart). Whether assist mode is used for any action isdetermined by the automation flag. This is checked to see whether that action ispermitted.

Cases where you might want to use assist mode include:v During early stages of developing and using your automation policyv After changing your automation policy, such as after adding an application to

automationv After adding a new automation procedure to the SA z/OS code

Using Assist Mode to Test Automation ProceduresAssist mode can help you to detect problems with your automation proceduresbefore they are added to your production code. Assist mode works by interceptingcommands and replies before they are issued through NetView.

Installing Your Automation Procedures


|

|

The intercepted commands and replies, as coded in the automation policy, arereformatted into a message that is sent to the NetView log.

The reformatted command is issued in message AOF320I and the reformatted replyin message AOF323I. Each message contains detailed information about the actiondefined in the automation policy and the actual action to be issued.

During run time of SA z/OS, the assist mode can be enabled with the INGAUTOcommand to set the related automation flag to the value L. The DISPFLGScommand can be used to view the current automation flag settings. Any othervalue for the automation flag deactivates assist mode.

When an event triggers an automated action and assist mode is enabled, SA z/OSlogs the action in the NetView log. The log can be reviewed to ensure thatautomation has run as expected.

Assist mode works for all routines that call the SA z/OS automation routines, afterhaving checked the automation flag by calling AOCQRY.

Using AOCTRACE to Trace Automation Procedure ProcessingThe AOCTRACE command dialog maintains both global execution flow traces andautomation procedure-specific debugging flags.

Setting the global flag causes all routines that support tracing and all message IDsto record a statement in the NetView log whenever they are invoked. TheAOFDEBUG global variable is used to pass the global flag information to theautomation procedure. The global flag is set to null if the global trace is off, or Y ifthe global trace is on.

Setting the automation procedure-specific flags lets you obtain information aboutwhat the automation procedure is doing when it executes, or lets you activate aREXX trace. The debug flag is either null or Y, and is stored in theAOF.clist.0DEBUG common variable (where clist is the true automation procedurename).

The trace flag is set to null or a valid REXX trace type, as follows:v A (All)v R (Results)v I (Intermediate)v C (Commands)v E (Errors)v F (Failures)v L (Labels)v O (Off)v N (Normal)

The S (Scan) trace type cannot be used.

The trace flag is stored in the common global variable AOF. clist.0TRACE (whereclist is the true automation procedure name).

Message tracing can only be set from the command line, using the commandAOCTRACE MSG/id,ON|OFF where id is the message to be traced.

Testing and Debugging Automation Procedures


AOCTRACE is documented in IBM Tivoli System Automation for z/OS Operator'sCommands.

REXX Coding ExampleFor examples of code that can be placed at the beginning and end of your REXXautomation procedures to handle trace and debug settings, see AOFEXC00 in theSINGSAMP library.

When writing code to support the debug feature, you should expose loc. on allyour procedures and insert fragments of code to check the value of the loc.0debugflag and output relevant information. The loc.0me assignment makes theautomation procedure name available everywhere, so you can prefix all debugmessages with it. You can then tell where the messages are coming from. Forexample:

Myproc:Procedure expose loc.If loc.0debug = ’Y’ Then

’PIPE LIT /’ loc.0debug ’ has called procedure MYPROC/’,’| LOGTO NETLOG’

Return

NetView Testing and Debugging FacilitiesNetView provides several facilities to assist in testing and debugging automationprocedures.

To do detailed testing, you may want to trace every statement issued fromautomation procedures. This type of testing is enabled through the &CONTROLstatement for NetView command lists and through the TRACE statement for REXXprocedures.

You can also specify less detailed tracing on the TRACE and &CONTROLstatements, so that only commands are traced. A comparable facility, the interactivedebugging aid, is available for programs coded in PL/I and C.

Perform specific tracing by issuing NetView MSG LOG, PIPE LOGTO NETLOGcommands at appropriate points throughout a NetView command list, REXXprocedure, or PL/I routine.

To test for proper parsing and reaction to a message, write a short automationprocedure to issue a NetView WTO command. This WTO is processed by theNetView automation table and triggers the appropriate automation procedure. Ifthe automation procedure requires the job name, the job name must be temporarilyhard-coded to the appropriate name. In this case, because the WTO was issuedfrom the NetView region, the job name associated with the message is the NetViewregion. A sample automation procedure follows:WRITEWTO CLIST

WTO &PARMSTR&EXIT

The sample automation procedure can issue any single-line message by calling theroutine. For example, to issue message ABC123I, which indicates the start of aprogram, the command is:WRITEWTO ABC123I My testprogram PRGTEST has started.



Where to Find More Testing InformationMore information on testing can be found in the following books:v Tivoli NetView for z/OS Customization Guide

This book lists requirements for your programs, including preparing your codefor use, and detailed information on writing exit routines and commandprocessors.

v Tivoli NetView for z/OS Automation Guide

This book has guidelines for creating new automation procedures, including arecommended development process.

Coding Your Own Information in the Automation Status FileYou can code your own information in the automation status file with theASFUSER command.

The automation status file has 40 user data fields that are associated with eachresource that is defined within it. You may use these fields to store persistentinformation about resources that your code needs to access later. The informationin the ASF is not lost when SA z/OS is shut down. It lasts until one of thefollowing events occurs:v The ASF VSAM data set is deleted and redefined,v You bring SA z/OS up with an automation control file that does not include the

application that the information has been defined for

Note that you should verify that the information you have stored in theautomation status file is accurate whenever SA z/OS initializes, as circumstancesmay have changed while SA z/OS was down.

Each automation status file field reserved for your data can contain up to 20characters. The ASFUSER command allows you to update and display data inthese fields. See IBM Tivoli System Automation for z/OS Programmer's Reference forthe ASFUSER command description.

Programming RecommendationsThis section contains tips and techniques that may help to reduce the coding effortrequired when writing your own automation procedures, and to improveperformance of your automation procedures.v Use variables, such as &SUBSAPPL, &SUBSTYPE, and &SUBSJOB in place of

parameter values.To display output from a self-written command, prefix it with the procedurename. To be independent of a possible name change, use the 'parse source'REXX statement which provides the procedure name as the third token.Using NetView command list language variable JOBNAME for the resource fieldon an AOCQRY call, an automation procedure can be written to support aknown message for any job that can issue a message.

v Use defaults when possible to minimize coding.v Use generic error codes (see CDEMATCH).v Use available message parsing techniques:

– Use the NetView command PARSEL2R or REXX PARSE command to parse amessage without relying on a field position in a message.



|||

– Parse a message in the NetView automation table and send only necessaryfields to an automation procedure.

v Consider not coding the ENTRY field in CDEMATCH calls (default is theSUBSAPPL returned from the last AOCQRY call).

v Use appropriate automation flags.v Review the coding requirements in Tivoli NetView for z/OS Customization Guide

including restrictions to consider when writing code, such as:– Restrictions when TVBINXIT is on– Variable names– Macro use– Register use– Re-entering programs

v Use SA z/OS automation routines where possible, because they reduce yourmaintenance overhead.

v Use SA z/OS processor operations common commands where possible, becausethese:1. Are independent of the hardware type of the target system's processor2. Minimize the need for changes to your automation routines as you add new

processors to your enterprisev Consider using the NetView VIEW command to display online help text

associated with new code, and to develop a fullscreen interface for newcommands that are a part of the new code. Refer to Tivoli NetView for z/OSCustomization Guide for information on the VIEW command.

Global Variable NamesWhen creating your own automation procedures, you must ensure that the namesof any global variables you create do not clash with SA z/OS external or internalglobal variable names.

In addition, you must not use names beginning with:v CFGv AOFv INGv ISQv EVIv EVEv EVJ

Programming Recommendations


Global Variable Names


Chapter 3. Adding a Message to Automation

SA z/OS exploits the NetView automation table (AT) and message revision table(MRT). The AT contains traps for messages that must be automated. If an actionmust be taken in response to a message, this action needs to be defined in thecustomization dialog. A related AT entry is required to call a routine to execute theaction. The MRT allows you to modify message attributes such as color, routecode, descriptor code, display and syslog settings, and text of original z/OSmessages (rather than copies).

SA z/OS automatically generates the ATs, MRTs and the MPFLSTxx PARMLIBmember.

Conceptual OverviewThis section gives a brief overview of the main aspects of SA z/OS messageautomation:v Message automation is a process that is based on the NetView AT and MRT.v The ATs, MRTs and the MPFLSTxx PARMLIB member are generated by

SA z/OS.v AT entries are created for messages that actions are defined for.v Messages can be defined to indicate a status change.v Messages can be marked to be ignored, thus not generating an AT, MRT or MPF

entry.v Messages can be marked to be captured for further display.v Most AT entries trap messages independent of the issuing product instance,

component or module.v Predefined AT entries can be changed.v You can define the AT/MRT scope to determine precisely if and what kind of

AT or MRT is built.

Defining Actions for MessagesAT entries are generated by SA z/OS for messages that are defined for APL, MTR,or MVC policy entries and that have actions (for example, CMD or REP) definedfor them.

Note: Throughout this chapter, whenever the term policy entry is used, it implieseither an APL, MTR, or MVC policy entry, unless otherwise stated.

The first step in defining actions is to select a policy entry from the Policy Selectionpanel. From its policy selection list, select the MESSAGES/USER DATA policyitem. This leads to the Message Processing panel, where you can then defineactions for message IDs. If an AT entry is built according to the action, it onlychecks for the message ID by default, independent of the product instance,component or module issuing that message. If this is not intended, you can use theAT action (see “Defining Message Overrides” on page 28).

There are many messages that are already prepared by SA z/OS. For thesemessages specific AT entries are predefined by SA z/OS, see the


||

||

||

+SA_PREDEFINED_MSGS MVS component entry. If you want to know what kindof AT entry is built for automating a particular message, you can view it on theMessage Automation Overview panel.

You can use system automation symbols (AOCCLONE variables) or systemsymbols in message IDs. These symbols are resolved when AT and MRT areloaded, but cannot be resolved for MPF.

Defining CMD or REP ActionsSuppose, for example, that you define a CMD or REP action for message XYZ222Ion the Message panel, where XYZ222I is a completely new message that is notpredefined by SA z/OS.

This definition leads to the creation of an AT entry for message XYZ222I using theISSUEACT command after the next Configuration Build process.

Note: If you have code definitions that you expect to be passed to ISSUEACT, youhave to manage the AT overrides to do this. This is not done by SA z/OS. See“Defining Message Overrides” on page 28.

Note that for MVC entries, messages have the parameter SYSTEMMSG=YES added tothe SA z/OS command (ISSUEACT).

Defining AT ActionsYou can define various AT actions for messages using the Message AutomationOverview panel:v The condition in the AT entryv Status changes for messagesv Capturing messages to be displayed but not automatedv Preventing the building of AT entries

You can also edit the AT entries directly using the AO option from this panel. Notethat if you use one of the other options after you have specified an override,SA z/OS requires you to confirm whether you want to delete the override thatexists for the message because it cannot be combined with the other options.

Defining Conditions for AT EntriesYou can improve the efficiency of AT processing by controlling where entries areplaced within the AT and by specifying more precise conditions to trap themessage.

SA z/OS allows you to do this with the AT Entry Conditions panel, which youreach from the Message Automation Overview panel by entering the AC option.

Defining Status MessagesMany messages that indicate a state change of APL, MTR, and MVC resources areknown to SA z/OS. The related AT entries are already predefined.

For these messages there is no need to define them in the policy database.

If necessary, you can define additional application messages that indicate a statechange. The AT action leads to the Message Automation Overview panel, whereyou can enter the AS option to display the AT Status Specification panel that listsresource states.

Defining Actions for Messages


The Status Message Report shows all status messages. It lists all user-defined andpredefined status messages and their associated statuses.

Status messages can be defined for MVC policy entries as well as for APL andMTR instances or classes. As an example, to define an UP state indicated bymessage XYZ444I, enter A in the Cmd field next to the message ID on the MessageProcessing panel. On the Message Automation Overview panel, enter the ASoption to display the AT Status Specification panel and select the UP status. Here,XYZ444I is a message that is unknown to SA z/OS.

This definition leads to the creation of an AT entry for message XYZ444I using theACTIVMSG command after the next configuration build process, as shown on theMessage Automation Overview panel.

Notes:

1. There are certain messages that can be used as status messages, but for somemessages, COD definitions are required (for example, IEF450I). TERMMSG setsthe status depending on these definitions. For more details about TERMMSG,see IBM Tivoli System Automation for z/OS Programmer's Reference.The entries for message IEF450I entries are important. Sample policy *BASEincludes a set of recommended code definitions.

2. Automation table entries are generated based on the messages that are definedwith MESSAGES/USER data. For size and performance reasons, these entriesare message-oriented rather than job-oriented.This means that an AT action (except IGNORE) for a particular messagegenerates an AT entry. This entry traps that message independently of theissuing subsystem. It then sets the subsystem state as selected via the ATaction.If a state message should be processed for a particular subsystem only, you candefine an AT override action.

Defining Captured MessagesIf messages only need to be captured to be displayed but not automated, the ATStatus Specification panel provides an additional Capture option for APL and MVCentries.

Messages that have a CMD or REP action defined for them or that are defined asstatus message are implicitly captured. There is no need to explicitly define thesemessages to be captured.

For example, to define message XYZ555I to be captured, enter option AS on theMessage Automation Overview panel to display the AT Status Specification paneland select the Capture option.

This definition leads to the creation of an AT entry for message XYZ555I using theAOFCPMSG command after the next configuration build process, as shown on theMessage Automation Overview panel.

Inhibiting AT, MRT, and MPFLSTxx EntriesMessages that are marked IGNORE do not cause an AT entry, MRT entry, or anMPFLSTxx entry to be generated.

IGNORE discards other actions that are defined for the same message.


Chapter 3. Adding a Message to Automation 27

||

|

AT Entries That Are Never BuiltThere are many keywords that can be entered as message IDs in the customizationdialog (for example, message MVSDUMPFULL). No AT entry is built for thesekeywords.

A list of these keywords is given in the online help.

Defining Message OverridesYou can apply an override on the Message Processing panel for a message ID foran APL instance, APL class or an MVC entry.

The AO option on the Message Automation Overview panel allows you tooverride an AT entry. You can change any part of the AT entry. Condition andaction statements can be changed, added or deleted.

If you define a message with an AT action or condition, and then invoke theoverride panel, the preview of the AT entry is shown on the override editor screen.You can use this as a model for your own AT definition. Use the CANCELcommand to exit the editor without saving your changes.

Note that if you specify an AT status selection or an MRT action selection for amessage with an AT or MRT override then a confirmation panel for the "overridedelete" is displayed because an override cannot be combined with the otherspecifications.

You can include system automation symbols (AOCCLONE variables) and systemsymbols in an AT or MRT override definition. They are resolved at AT and MRTload time.

You can define '&*JOBNAME.' as part of an AT condition statement that will bereplaced by the jobname of the given policy entry when building the AT. This isvery valuable when defining an AT entry for an APL class. Then each APL instancelinked to that class will have its own AT entry with its jobname in the ATcondition statement. Checking for the jobname may also be required if differentinstances of a product issue the same message but you only want certain jobs to beaffected by that message.

To define an override for message XYZ666I, for example, on the MessageProcessing panel, you can either change an existing AT entry that then becomes auser-defined AT entry, or, if no predefinitions are available, you can define auser-specific AT entry.

Extended Status Command SupportThe status command concept has been extended so that commands can be issued iftwo linked or dependent applications reach an “up” or “down” state.

Thus, a command can be issued for one application (APL1) when another oneAPL2 enters or is in a certain state. Application APL1 is a consumer that consumesservices provided by application APL2 which is a provider. In certain cases it isvaluable to trigger a consumer action if the provider enters an UP or DOWN state.Alternatively the consumer may take an action when coming up, which dependson the state of the provider.



In addition to runtime variables such as &SUBSAPPL or &SUBSJOB, there areprovider-specific runtime variables that can be used within a command or reply asspecified for a Message ID in the MESSAGES/USER DATA policy. These variablesstart with &SU2 instead of &SUB. For a list of supported provider runtimevariables, see ACFCMD and ACFREP in IBM Tivoli System Automation for z/OSProgrammer's Reference.

Policy DefinitionsLinks must be defined in the APL policy for the consumer application.

A link is a data pair that is represented by a consumer and a provider subsystemname:v The consumer name is defined as the Subsystem Name of the consumer APLv The provider name is defined in the MESSAGES/USER DATA policy item of the

consumer APL as part of a pseudo Message ID

In order to execute a command by a consumer APL whenever a provider APLbecomes UP or DOWN, use:v UP_provider-subsys - to hold the commands that are executed when

provider-subsys becomes UPv DN_provider-subsys - to hold the commands that are executed when

provider-subsys becomes DOWN.

A dynamic link is created by defining a USER action with the Keyword/Data pairDYNAMIC=YES for the pseudo message ID.

Other pseudo messages are available to hold sets of commands on a consumerAPL that are executed whenever the consumer becomes UP and the provider APLis UP or DOWN respectively:v ISUP_provider-subsys - to hold the commands that are executed when the

provider-subsys is UPv ISDN_provider-subsys - to hold the commands that are executed when the

provider-subsys is DOWN.

No commands are issued when the consumer becomes DOWN. For dynamic linksthe commands are executed when the link is activated and the provider is in theappropriate status.

For example, if you want to start the MQ Listener for the MQCHIN applicationwhenever the TCPIP application reaches an up state, you define a command forthe message ID UP_TCPIP using the MESSAGES/USER DATA policy item ofMQCHIN. On the Message Processing panel, enter the line command C for theUP_TCPIP message ID and on the subsequent Command Processing panel enterMVS START LISTENER in the Command Text field.

For dynamic links, you need to define message IDs for all the providers that mightbe used by a consumer. There are several alternatives for defining when a link isactivated:1. In the POSTSTART phase of the STARTUP policy item2. Based on a user-selectable message before the consumer is in the UP state3. Based on the “up” state (this can be UP or ENDED)4. Based on a user-selectable message after the consumer is in the UP state



You can make these definitions in either the STARTUP or MESSAGES/USERDATA policy item. Corresponding policy definitions are required for linkdeactivation either in the SHUTDOWN or MESSAGES/USER DATA policy item.

You must supply a user-written REXX automation procedure that:1. Identifies the provider2. Issues the INGLINK command to activate the link, if required

All the dynamic links of a consumer are automatically DEACTIVATED after theconsumer is shut down. Thus you only need to define DEACTIVATE commands ifyou want this process to happen at an earlier point in time. Furthermore, you mustnot define a DEACTIVATE if the consumer is its own provider since thisDEACTIVATE may take place before the DN_ message can be processed.

There is no difference in the DN_ and UP_ message processing between normaland transient applications, with one exception: when stopping a transient providerthe corresponding DN_ message on the consumer is not processed.

Special ConsiderationsYou can define a consumer as its own provider. This can be done to perform apredefined action for an application whenever it enters any “down” state. A“down” state in this context is an agent state of:v DOWNv RESTARTv AUTODOWNv STOPPEDv CTLDOWNv BROKEN

Thus you no longer need to define the same action for each of the agent states. Toachieve this, define the same name in the DN_ message and subsystem name fieldof an application policy entry. You should note, however, that:v If the consumer and provider applications are different, the action defined under

DN_ is only executed, when the consumer application is in the “up” state.However if the consumer and provider applications are the same, the actiondefined under DN_ is executed even when the consumer application is not inthe “up” state.

v Dynamic link definitions are not required. If you define a dynamic link, it isignored.

v The action defined for the DN_message is not executed if the status changeresulted from a failed startup.

Defining Entries for the Message Revision TableThe message revision table (MRT) enables user-defined modification of attributessuch as color, route code, descriptor code, display and syslog settings, and text oforiginal z/OS messages (rather than copies).

You can make decisions about the message based on its message ID, job name, andmany other properties. You can have only one MRT active per system.

Any MRT specifications that you make are independent of any AT entry data forthe message. Thus if you make a definition for the MRT but not for the AT, anyexisting data for AT entry is still in effect.



The MS option on the Message Automation Overview panel allows you to defineconditions and attributes that are used to generate NetView message revision table(MRT) entries for a message ID. Use the options on the Message Revision TableConditions panel to specify the following:v Delete the message completely (if you select this, no other selection is valid)v Whether to automate the messagev Suppress the message from the console or system logv Translate the message text to uppercase or append further text to itv Change the color, highlighting, and intensity (if your terminal supports high

intensity) attributes of the message. Only one selection for each of theseattributes is allowed.

A syntax check is not made of the MRT entry because any system-specificdefinitions (for example, symbols) can only be verified on the system where theMRT is to run.

You should check that routines that are triggered by the AT entry for the messageID are compatible with any text that you append to the original message text.

You can also use the MO option on the Message Automation Overview panel todefine the MRT entry directly using a fullscreen editor. Note that a syntax check isnot performed on this panel. You must ensure that your specifications followNetView message revision table syntax rules.

For more details, see the chapter “The Message Revision Table” in IBM TivoliNetView for z/OS Automation Guide.

BuildOnce you have made all the message definitions that you need, you can start theConfiguration Build Process to build the configuration files containing the AT,MRT, and MPF table.

For more information about the build function, see the chapter “Building andDistributing Configuration Files” in IBM Tivoli System Automation for z/OS DefiningAutomation Policy.

The AT fragments, MRT, and the MPFLSTxx member are built into theconfiguration data output data set.

This may require more space than you have allocated for the output data set. Thusenlarging the output data set may be required.

This also applies to the DSILIST data set where the listings are stored.

It is recommended that you copy the build output to a Generation Data Group(GDG) to avoid token mismatch conditions and AT or MRT load errors.

AT and MRT Build ConceptThe AT and MRT are built if necessary.

Note: If the MPF Header or Footer definitions have changed, an MPFLSTxx buildis not performed. The changes are taken into account at the next build.



LoadAfter the NetView automation tables have been generated using the customizationdialog, they are ready to be loaded.

INGAMS REFRESH can be used to refresh the complete SA z/OS configuration,that is, the Automation Manager Configuration (AMC), the agent's AutomationControl Files (ACFs) and the related NetView Automation Tables (ATs) as they aredefined in the SA z/OS Policy Database. Alternatively, ATs can be loaded usingATLOAD.

The common global variable AOFSMARTMAT controls whether the AT and MRTfragments generated at Automation Control File build should be used. Forcompatibility reasons the provided default is 2 indicating that the generated ATfragment is loaded at SA z/OS initialization time or during an INGAMSREFRESH. The recommended value of AOFSMARTMAT is 3 indicating that thegenerated AT and MRT fragments are loaded at SA z/OS initialization time orduring an INGAMS REFRESH.

For more information about the values refer to AOFSMARTMAT in “Read/WriteVariables” on page 242.

Some AT entries are required for SA z/OS to operate properly. These entries residein a separate AT that is loaded during SA z/OS initialization. This AT is calledINGMSGSA. Do not edit it.

ListingsThe DSILIST data set is used to store listings.

For example, if you want to view the listing of the AT INGMSG01, issue thecommand:br dsilist.ingmsg01

To view the listing of the MRT, issue:br dsilist.ingmrt01

A listing is produced whenever SA z/OS loads an AT or MRT. You can use theadvanced automation option (AAO) AOFMATLISTING to suppress listings bysetting it to zero (see “Global Variables” on page 199).

The AT can be reloaded at configuration refresh (INGAMS, ACF ATLOAD).Because of this you should:v Use a separate DSILIST data set for each NetViewv Allocate the DSILIST data set as a PDSE in order to prevent Sx37 errors

If the testing or loading of an AT or MRT fails, a special INGERRLS listing thatcontains the data of the failing AT or MRT is written to DSILIST. To view thislisting issue the following command:br dsilist.ingerrls

Load


A Guide to SA z/OS Automation Tables

NetView Automation Table StructureSA z/OS provides a ready-to-use AT, INGMSG01.

To activate the AT, perform the following steps:1. Define the AT member INGMSG01 in the SYSTEM INFO policy of the system

in the customization dialogs2. Build the automation configuration files3. Refresh the configuration using INGAMS REFRESH4. Restart NetView with the new configuration

The SA z/OS AT contains:v All entries for the SA z/OS basic automation infrastructure, which reside in

INGMSGSAv AT entries for messages that are defined in the PDBv User include fragments

You do not have to customize the AT INGMSG01. All unused entries are disabledautomatically according to the configuration that you use. If you want to haveadditional entries that are valid only for your environment, you can use either aseparate AT (specified in the customization dialog) or use one of the user includes.

Figure 5 shows the structure of the AT:

For information about how to use the INCLUDE fragments that SA z/OSprovides, refer to “Using SA z/OS %INCLUDE Fragments” on page 34.

The following fragments are used by the AT:

Synonym DefinitionsThere is one fragment, AOFMSGSY, that is used to initialize the varioussynonyms used throughout the rest of the table. SA z/OS requires thesynonyms to be suitably customized to reflect your environment. See“Generic Synonyms: AOFMSGSY” on page 283 for more details about thesynonyms.

SA z/OS Functional DefinitionsThese definitions (located in the fragment that is loaded as INGMSG02)contain automation table statements for specific functions of SA z/OS. Youshould not change these statements. Any modifications can be made inINGMSGU1.

INGMSG01│││──── %INCLUDE AOFMSGSY││──── %INCLUDE INGMSGU1││──── %INCLUDE INGMSG02 (auto-generated)│└──── %INCLUDE INGMSGU2

Figure 5. AT Structure

A Guide to SAz/OS Automation Tables


Master Automation TablesThis section discusses the three master automation tables that SA z/OS provides.

INGMSGSA:SA z/OS provides this automation table containing all statements that are requiredfor SA z/OS to work properly.

INGMSG01:INGMSG01 is suitable for use as a primary automation table.

INGMSG01 should not be included into any other table but should be activated asa separate table.

AOFMSGST:This is a table suitable for a NetView with a SA z/OS Satellite installed.

Integrating Automation TablesIf you have any user-written automation table statements that you still want touse, you must now combine your primary table with the automation table forSA z/OS. There are several approaches to achieve this.

Refer to the NetView documentation for more information on how to use NetViewautomation tables.

Multiple Master Automation TablesBesides INGMSG01, you can specify multiple additional NetView automationtables for a system in the customization dialog. The tables are concatenated asentered in this panel and processed in this concatenation order.

You need not modify the INGMSG01 automation table or any of the fragments,except AOFMSGSY. It is easy to maintain SA z/OS automation table fragments.However, you have to watch for new messages. It is easy to maintain your entries,because they are independent from SA z/OS entries.

Using SA z/OS %INCLUDE FragmentsINGMSG01 is the master include member. It provides some message suppressionthat is necessary to prevent mismatches and duplicate automation before the first%INCLUDE fragment.

The INGMSGU1 fragment can be used for user entries. These entries haveprecedence over the SA z/OS entries. The default INGMSGU1 fragment is anempty member.

The INGMSGU2 fragment can be used for all entries that SA z/OS does notprovide any entries for. The default INGMSGU2 fragment is an empty member.During ACF COLD or WARM start the AT or are loaded and they write a listing tothe DSILIST data set. This enables the use of the NetView AUTOMAN commandto monitor and manage the ATs. Make sure that the size of your DSILIST data setis sufficient to store these listings. Without these listings you can only monitor ormanage the ATs using AUTOTBL. It is recommended that you define your DSILISTdata set as a PDSE so that regular data set compression is not required. You shouldalso make sure that the DSILIST DSN is unique to your NetView procedure.

Examples:An example output of AUTOTBL STATUS:

NetView Automation Table Structure


|

|||

BNH361I THE AUTOMATION TABLE CONSISTS OF THE FOLLOWING LIST OF MEMBERS:AUTO2 COMPLETED INSERT FOR TABLE #1: INGMSG01 AT 04/16/02 19:34:59AUTO2 COMPLETED INSERT FOR TABLE #2: HAIMSG01 AT 04/16/02 19:35:00

IPSNOBNH363I THE AUTOMATION TABLE CONTAINS THE FOLLOWING DISABLED STATEMENTS:TABLE: INGMSG01 INCLUDE: __n/a___ GROUP : INGCICSTABLE: INGMSG01 INCLUDE: __n/a___ GROUP : INGIMAGETABLE: INGMSG01 INCLUDE: __n/a___ GROUP : INGIMSTABLE: INGMSG01 INCLUDE: __n/a___ GROUP : INGJES3TABLE: INGMSG01 INCLUDE: __n/a___ GROUP : INGOPC

An example of the AUTOMAN panel:

EZLKATGB AUTOMATION TABLE MANAGEMENT

MEMBER TYPE LABEL/BLOCK/GROUP NAME(S) STATUS NUMBER OF STATEMENTS-------- ----- ------------------------- -------- --------------------INGMSG02 GROUP INGCICS DISABLED 222INGMSG02 GROUP INGDB2 ENABLED 120INGMSG02 GROUP INGIMAGE DISABLED 1INGMSG02 GROUP INGIMS DISABLED 107INGMSG02 GROUP INGJES2 ENABLED 1INGMSG02 GROUP INGJES3 DISABLED 1INGMSG02 GROUP INGOPC DISABLED 10INGMSG02 GROUP INGUSS ENABLED 1

In this example the configuration loaded does not use the IMS, CICS®, OPCproduct automation and the IXC102A automation. It uses JES2, DB2 and USSautomation.

Restriction:The NetView AUTOMAN cannot be used to RELOAD INGMSG01.

Generic Automation Table StatementsThe basic automation table contains a number of generic automation table entriesthat can reduce your automation table overhead considerably. These samples usesome of the advanced features of SA z/OS to make automating your applicationsas simple and reliable as possible.

For some of these entries (IEF403I and IEF404I in particular) the message flow maybe quite high. To handle this, you can insert additional entries in INGMSGU1 tosuppress a block of messages. For example, if all your batch jobs started with thecharacters BAT or JCL, then the following entry would suppress them:IF MSGID = ’IEF40’. & DOMAINID = %AOFDOM% THEN BEGIN;*

IF (TOKEN(2) = ’BAT’. | TOKEN(2) = ’JCL’.)THEN DISPLAY(N) NETLOG(N);

*END;

System Operations Automation FlowSA z/OS uses dedicated work operators for all subsystem-related processing inorder to:v Keep the extra message-related workload off the NetView subsystem interface

router task (CNMCSSIR) and primary POI task PPTv Establish even load balancing

Integrating Automation Tables


v Ensure that all messages for a subsystem are processed in the correct sequence

You define work operators in the customization dialog using the AutomationOperators entry type (AOP). Here you define automated functions that allow youto specify automation operators, which are the NetView task name. Thistwo-staged definition gives the flexibility to specify a second operator as a backupwithin the same Automated Function definition. The automation operator namethat is specified here is the name of the task in NetView.

Note that the automation operators also need to be defined in the DSIOPF memberin the NetView DSIPARM data set or in the SAF product.

By default SA z/OS provides 20 Automated Functions, AOFWRK01 throughAOFWRK20, with the automation operator names AUTWRKxx. The number canbe increased according to the installation needs.

During SA z/OS initialization or refresh the subsystems that are defined in theconfiguration file are evenly distributed among the automation operators in around-robin manner. Thus each automation operator has a list of subsystems that itis responsible for. Each automation operator then subscribes for the messages ofthose subsystems via the NetView ASSIGN command. Finally the initialmonitoring of SA z/OS is run on the appropriate automation operator, which isthen locked until message AOF540I is issued.

When SA z/OS is fully initialized all messages for a subsystem are queued to thesame automation operator. This ensures that all messages are processed in theorder they have been received.

If the automation table action uses standard SA z/OS capabilities (that is,SA z/OS commands), the message is processed at the automation operator in thefollowing three steps. However, if there is a complete user defined automationtable entry (that is, an AT override), only the first step can be run:1. The message is driven through the NetView automation tables.2. If there is a match, the SA z/OS data model is applied, which includes

automation flag checking, code matching, threshold comparison, passevaluation, and message capturing.

3. Finally the command is executed or the outstanding reply is answered.

There are two places where this processing can be modified for single messages:v The assignment of messages to AUTWRKxx automation operators can be

overruled.To do this, the AOF_ASSIGN_JOBNAME advanced automation option must beset to 0, which lets ASSIGN BY MESSAGE ID take precedence over the ASSIGNBY JOBNAME that is established by SA z/OS.An ASSIGN command with the MSG parameter must be issued to redirect themessage. That particular message is then assigned according to the userspecification while all other messages still run on the automation operators thatare assigned by SA z/OS. However this should be used with care because itsuspends SA z/OS load balancing and breaks the serialized commandprocessing for that subsystem.

v Execution of the command on the automation operator that has been assignedby SA z/OS can be overruled by specifying an Automated Function nametogether with the command in the MESSAGES/USER DATA policy in thecustomization dialog.

Generic Automation Table Statements


Execution of the command is then routed to the task that has been specified forthe Automated Function. The automation table and data model processing is stillrun on the automation operator and thus proper sequencing is guaranteed.SA z/OS internally uses the AOFEXCMD command (described in IBM TivoliSystem Automation for z/OS Programmer's Reference) to queue the command to thespecified automation operator. The routine checks whether the requestedautomation operator is available and, if this is not the case, it queues thecommand to a backup operator, so that in any case the command does not runon the current automation operator.It is recommended that you use this only if there are special reasons, forexample, for long running commands, because it may break the serializedcommand processing for that subsystem (if not all commands are executed onthe same automation operator).

Inheritance Rules for ClassesBear in mind the following inheritance rules for class data when building ATentries.

Define Application InformationData is inherited in the APPLICATION INFO policy item per individual field,independent from each other (except for Transient Rerun).

If a field is blank, the class value is inherited (if it is available). There are a fewexceptions where the inheritance can be blocked without specifying an instancevalue with the special value NONE , for example, Restart after IPL.

Define RelationshipsThe External Startup and External Shutdown fields in the sub header area showinherited data individually in the same way as in the APPLICATION INFO policyitem.

However the relationships are only inherited as a whole if no relationships aredefined for the child object. It means that on the relationship panel, it's either allinherited data or no inherited data at all. As soon as inherited data is modified ornew relationship data is added on the child level, all relationships are consideredchild (instance) definitions with no relationship inheritance at all. The only way togo back to class inheritance is to delete all the relationships on the instance level.

Define Application Messages and User DataData is inherited per message ID.

For example, assume a message ID has a command definition for the instance, andthe same message ID is defined for a class with reply data. The command andreply data is not merged on the instance, and the class definitions are not inheritedat all. Message overrides (OVR) are not inherited at all. All OVRs are used togenerate AT entries at the level where they are specified.

Define Startup ProceduresThe STARTUP policy offers two panels. The Subsystem Startup Processing panelwith a subheader section with input fields that may show inherited values, and foreach selected startup phase there is a Startup Command Processing panel with acommand input area that also may show inherited data.



Subsystem Startup ProcessingData in the subheader section is inherited per individual field, similar to theAPPLICATION INFO policy. Command definitions for the different phases areinherited per start phase. So if PRESTART commands are defined for the instanceand both PRESTART and STARTUP commands are defined for a class, the instanceinherits the STARTUP commands from the class.

Startup Command ProcessingWithin each start phase the commands are inherited all together. So if a PRESTARTcommand is defined for the instance and other PRESTART commands are definedfor a class, none of the commands are merged on the instance. Instead the instancehas only the one command defined there. No PRESTART commands are inheritedfrom the class.

Define Shutdown ProceduresShutdown specifications are inherited per phase. So if a SHUTINIT command isdefined for the instance and both SHUTINIT and SHUTNORM commands aredefined for a class, the instance inherits the SHUTNORM commands from theclass.

Furthermore, command and reply definitions for one phase are inherited together.So if for SHUTFORCE a command is defined for the instance, and the class has areply defined for SHUTFORCE, nothing is inherited by that instance.

Changes within inherited data result in creating definitions for the currentapplication. So if for a phase, commands and reply definitions are inherited, andthen commands are modified, both the reply and the command definitions becomedata of the current application. If only commands are inherited for a phase, andthen reply data is specified, the command definitions are also copied to the phasedefinition of the current application.

Define Error ThresholdsThe data is inherited as a whole if no thresholds are defined for the child object –it is not possible to specify a level for Critical, Frequent, or Infrequent alone for aninstance and inherit the other threshold levels from a class.

Define IMS Subsystem-Specific DataThis policy combines fields that are built into the IMSCNTL and theENVIRONMENT structures of the configuration files. The fields within a structureare inherited all together, but each structure is inherited independently from theother.

Furthermore the IMSCNTL fields do not allow definitions for a class (though theyare displayed on the class panel). And finally for a subtype other than CTL only asubset of the fields is available.

Thus there are three variations of this panel:1. Instance of subtype CTL with all IMSCNTL and all ENVIRONMENT fields2. Class of subtype CTL with all ENVIRONMENT fields3. Instance or Class of subtype other than CTL with a subset of ENVIRONMENT

fields (2 fields)



|||||

The first four fields (APPLid, Default HSBID, Startup parm1, Startup parm2) arenever inherited. They cannot be specified for a class. The remaining fields areinherited all together in a blocks.

Automatic AT GenerationCMD (Command), REP (Reply), COD (Code), and USR (User Data) are inheritedper message ID. For example, assume a message ID has a command definition onthe instance, and the same message ID is defined for a class with reply data. Thecommand and reply data are not merged on the instance, and the class definitionsare not inherited at all.

Messsage Overrride and Status specifications provide instructions for thegeneration of the AT entry. This data is never inherited, but is used to create oneAT entry for the object where they are specified. Remember that the AT is messageoriented and the AT entry usually has the message ID as a condition, so forexample, inheriting a Status would create duplicate entries.



Chapter 4. Monitoring Applications

System Automation for z/OS provides different ways to monitor your applications:v Using observed status monitoring routines, SA z/OS can determine whether your

applications and several other automated resources are active, inactive, or in theprocess of being started. It is recommended to always enable observed statusmonitoring routines and to use the product-provided routines where possible.See “Observed Status Monitoring” for further details.

v With monitor resources you can optionally monitor the health of your applicationsand recover them on health status changes. SA z/OS distinguishes betweenactive health monitoring and passive event-based health monitoring. See “HealthMonitoring” on page 42 for further details.

Active and passive health monitoring is supported by SA z/OS in the followingareas:v Health monitoring of JES3, based on console messagesv Health monitoring of z/OS, DB2, CICS, IMS and other components, based on

IBM Tivoli OMEGAMON II exceptions or IBM Tivoli OMEGAMON XEsituations

v Health monitoring of CICS, based on CICSPlex® SMv Health monitoring of IMS, based on console messages

Observed Status MonitoringSA z/OS determines the observed status of an application by running a routineidentified by the policy administrator in the customization dialog.

The routine can be specified for an individual application (refer to IBM TivoliSystem Automation for z/OS Defining Automation Policy), and a default monitorroutine can be specified for all applications on an entire system (see theAUTOMATION INFO policy item in the customization dialog).

Table 5 lists the routines that can be specified as application monitors.

Table 5. Observed Status Monitor Routines

MonitorRoutines Using This Routine

AOFADMON This routine has been deprecated and is provided only for compatibilitywith earlier releases. This routine determines the status of an applicationby issuing the MVS D A, jobname command. The job name used is thejob name defined in the customization dialog

AOFATMON This routine is used to determine the status of a task operating withinthe NetView environment.

AOFAPMON This routine determines the status of a program-to-program interface(PPI) receiver.

AOFCPSM This routine is a dedicated routine used to monitor the status of theSA z/OS processor operations applications.

AOFNCMON This routine is used to determine the status of the NETCONVconnection running between the NMC server and NetView for z/OS.


Table 5. Observed Status Monitor Routines (continued)

MonitorRoutines Using This Routine

AOFUXMON This routine determines the status of a resource with application typeUSS. This resource can either be a z/OS UNIX process, a file system inthe UNIX file system (HFS), or a TCP port. Depending on the nature ofthe resource (process, file, or port) AOFUXMON decides which internalmonitoring method to use.

INGPJMON This routine determines the status of an application by searching z/OSfor address spaces with a particular job name. The job name used is thejob name defined in the customization dialog for the application.

INGMTSYS With this routine, IMAGE applications for BCPII usage can bemonitored.

INGROMON With this routine, OMVS can be monitored.

INGPSMON This routine monitors the subsystem's registration to the subsysteminterface.

ISQMTSYS With this routine, a processor operations target system resourcerepresented by its proxy can be monitored. See “Automating ProcessorOperations Resources of z/OS Target Systems Using Proxy Definitions”on page 91 for examples of how to use a proxy definition. Activeoperator console connections are mandatory and are used for sending az/OS command (for example, d t) and receiving the related response.

SA z/OS expects certain return codes from all monitor routines, either fromSA z/OS provided ones or from your own routines. These can be one of thefollowing:

RC Meaning0 Active4 Starting8 Inactive12 Error

Note: When you write your own monitor routine, you must consider that thisroutine will also be executed during initial status determination. This processoccurs prior to common global variable AOFCOMPL is set to YES and messageAOF540I is issued, indicating INITIALIZATION RELATED PROCESSING HASBEEN COMPLETED.

Health Monitoring

OverviewHealth monitoring is accomplished using special resources called monitor resources.Monitor resources, which have a resource type MTR, are policy objects that areused to obtain the health status of other resources, typically applications orapplication groups, or more generally, any object that can be monitored. The healthstatus is useful when you need to know how well a resource is performing and notsimply that it is active.

The health status can be used to provide application-specific performance andhealth monitoring information, for example, an application may be active but it isfailing to meet performance objectives defined by the system administrator. The

Observed Status Monitoring


health status can be used either for information only, or by the automationmanager to make decisions and, if necessary, trigger automation for theapplication.

Monitor resources are defined in the customization dialog with entry type MTR.They are resources with similar characteristics as all other SA z/OS resources.

Monitor resources are connected to application resources (APLs) or applicationgroup resources (APGs). The health status of the monitored object is propagated tothe APLs and APGs and results in a combined health status there. You can defineand connect MTRs in the customization dialog (see IBM Tivoli System Automationfor z/OS Defining Automation Policy).

Monitor resources obtain the health status of an object in two different ways:v Actively, by polling—that is executing a monitoring command periodicallyv Passively, by processing events

Active monitors are scheduled periodically based on the interval defined in theMTR policy.

Passive monitors do not have a monitor interval but can have a monitor commanddefined for them for initial health status determination. They rely on other eventsto set the health status using the INGMON command.

Monitor resources can be explicitly bound to the object that they are monitoringand optionally to a job. This allows SA z/OS to handle a variety of monitoringevents in a generic way. A monitored object can be, for example, an OMEGAMONXE situation, or an event posted by CICSPlex System Manager (CICSPlex SM). See“Passive, Event-Based Health Monitoring” on page 46. Note that the monitoredobject is derived from the monitor resource name, if none was specified.

There can be one or more recovery commands associated with each health status(NORMAL, WARNING, MINOR, CRITICAL and FATAL). These commands areinvoked by SA z/OS when the monitor resource switches to the correspondinghealth status.

You can display and control monitor resources with the DISPMTR command.Monitor resources are also displayed on the Tivoli Enterprise Portal (TEP) as wellas SDF and NMC, provided that the appropriate inform list specifications havebeen made.

Monitor Resource CommandsWhen defining a monitor resource you can specify activate, deactivate and monitorcommands. Any command is suitable that can be executed in the NetViewenvironment. These commands are divided into two groups:v NetView activate and deactivate commands that expect a return code of zerov Monitor commands that return a health status

The main difference between these two groups is that the activate and deactivatecommands are executed only once, and SA z/OS expects a return code of zero.

If the activate command ended with a non-zero return code, the monitor resourceremains in an INACTIVE status. The monitor resource ends in a BROKEN status ifthe deactivate command ended with a non-zero return code.

Health Monitoring

Chapter 4. Monitoring Applications 43

v The activate command is optional and can be used to establish the environmentthe monitoring routine can run in. The command is executed every time themonitor is started. The command must exit with return code 0.

v The deactivate command is optional and can be used to cleanup theenvironment. The command is executed every time the monitor is stopped. Thecommand must exit with return code 0.

v The monitor command is executed after the activate command and thenperiodically if a monitoring interval is given. SA z/OS expects the monitorcommand to return a valid health status code. Additionally the monitorcommand can issue a message that is then attached to the health status. Theabsence of a monitoring interval indicates that the given monitor resource is apassive or event-based health monitor. In this case, the monitor command isoptional and, if specified, it is invoked for initial health monitoring only.Otherwise, if a monitoring interval is provided, the given monitor resource is anactive health monitor. In this case, a monitor command must be provided toreturn a health status.

The activate, deactivate and monitor command can be a command procedurewritten in any language that is supported by NetView: REXX, Assembler, PL/I, C,or the NetView Command List Language (NCLL). Writing a monitor routine canbe simple or it can be complex. The complexity depends upon the application thatyou are attempting to monitor.

Writing a Recovery RoutineThe recovery routine is invoked every time the monitor resource switches to thehealth status that the recovery routine is defined for. The goal of the recoveryroutine is to bring the monitor resource, and thus the monitored object, back to ahealth status of NORMAL.

Recovery TechniquesUser data in the MESSAGES/USER DATA policy item can be used to disableadditional recovery processing while other recovery is already in progress.

In combination with the predefined keyword DISABLETIME, the recovery disabletime can be specified in the formats hh:mm:ss, mm:ss, :ss, or mm. While recoveryis disabled, no commands are processed on behalf of this monitor resource formessages and exceptions that are specified in the MESSAGES/USER DATA policyitem.

Recovery is automatically enabled after the recovery disable time has expired.Recovery can also be enabled prematurely by calling the INGMON command withthe option CLEARING=YES, for example:INGMON CI2XREP MSGTYPE=XREP CLEARING=YES

In some cases, it is necessary to force increasingly strong recovery actions over aperiod of time. This can be accomplished using a PASS count that starts at 1 andruns to 99. SA z/OS maintains the PASS count individually per message orexception, and increments the PASS count each time that message or exception isprocessed. Upon successful recovery, it is the installation's responsibility to resetthe PASS count. When specified with option CLEARING=YES, INGMON enablescommand processing for messages and exceptions, and resets the PASS count.

Health Monitoring


Task Global Variables for Recovery RoutinesThe following task global variables can be accessed by the recovery routine:

Task Global Variable Value

&EHKVAR1 Contains the monitor name

&EHKVAR2 Contains the current health status

&EHKVAR3 Contains the old health status

&EHKVAR4 Contains the message that is associated with the health status

&EHKVAR5 Contains the object name of the monitor

&EHKVAR6 Contains the job name

&SUBSAPPL Contains the monitor name

&SUBSTYPE Contains the string MONITOR

Active Health MonitoringIn general, the monitor command needs to issue one or more commands togenerate data, process the data, and set a return code. The return code is then usedby SA z/OS to determine the health status for the resource.

The possible return codes and the corresponding health status are given in Table 6.

Table 6. Health Status Return Codes

Return Code Health Status Description

1 BROKEN The monitor detected an unrecoverable error.SA z/OS stops monitoring.

2 FAILED The monitor is currently unable to obtain a healthstatus. SA z/OS keeps the monitor active becausethe problem might disappear.

3 NORMAL The monitor detected normal operation of themonitored object.

4 WARNING The monitor detected a certain degree ofdegradation in the operation of the monitored object.

5 MINOR The same as WARNING, but more severe.

6 CRITICAL The same as MINOR, but more severe.

7 FATAL The same as CRITICAL, but more severe.

8 DEFER Used internally.

The health status values affect the compound status in the automation manager.

Most monitor commands use UNKNOWN, NORMAL, and WARNING statuses.The MINOR, CRITICAL, and FATAL statuses can be used as gradients to indicatethat a problem is getting worse. BROKEN and FAILED are statuses that describethe status of the monitor itself and may be seen if an error is encountered with themonitor command. A health status of FATAL will trigger an application move aspart of automated recovery.

FATAL is a guaranteed automatic ForceDown, and, if available, failover for theapplication associated with the monitor.

Health Monitoring


Optionally, the monitor routine can issue a message describing the condition that istrapped by the SA z/OS process that invoked the monitor. The message can beviewed on the DISPMTR panel.

Every monitor command needs several basic steps:1. Issue one or more commands to collect data and interrogate the results.2. Based on the results from the command or commands, set the return code to a

value from 1 through 8 and, optionally, perform processing based on thatvalue.

3. Optionally, supply more descriptive information about the health status in amessage that can be viewed with the DISPMTR command.

4. Exit with the return code so SA z/OS can set the health status appropriately.

Figure 6 is an example using the NetView PING command within a PIPE to querythe status of a TCP/IP stack on a remote system. The IP address is passed oninput. The routine uses the average round trip time (RTT) for the request providedin message BNH770I to determine the health.

Passive, Event-Based Health Monitoring

OverviewPassive, event-based monitoring allows you to react to events, for example amessage, an OMEGAMON XE situation, or a CICSPlex SM event, directly. Incontrast to active health monitoring, SA z/OS does not have to query themonitored object status periodically but is informed only when such an event hasoccurred.

The definitions in the MONITOR INFO policy item for a monitor resource allowyou to define an object that the monitor resource is bound to and optionally a jobthat the monitor resource accepts events from.

The Monitored Object specification for the monitor resource can follow anynaming convention that might be required for the monitoring process. For

/*REXX MYMON */Arg parmmonrcs=’BROKEN FAILED NORMAL WARNING MINOR CRITICAL FATAL DEFER’’PIPE (STAGESEP | NAME PING)’,’| NETV PING’ parm,’| LOCATE 1.8 /BNH770I /’,’| STEM out.’if out.0 = 0 then

lrc = wordpos(’FATAL’,monrcs)else

doparse var out.1 . ’averaging’ ms ’ms’ .say ’PING lasted’ ms ’ms’select

when ms < 10 then lrc = wordpos(’NORMAL’,monrcs)when ms < 20 then lrc = wordpos(’WARNING’,monrcs)when ms < 30 then lrc = wordpos(’MINOR’,monrcs)when ms < 40 then lrc = wordpos(’CRITICAL’,monrcs)otherwise lrc = wordpos(’FATAL’,monrcs)

endend

Return lrc

Figure 6. Sample Monitor Command

Health Monitoring


example, for CICS monitoring it has the prefix CPSM, followed by the CICS name,the type (such as a connection), and the name. For a link called CT12, themonitored object is called as follows, for example:CPSM.CICSTOR1.CONNECT.CT12.

Whereas for monitoring OMEGAMON XE situations, it has the prefix ITM,followed by the situation name, for example: ITM.MYAUXSHORTAGE_WARN.

There can be only one monitored object per monitor resource but more than onemonitor resource can be bound to a monitored object, for example, several IMSmonitors might specify OLDS as an object.

You can also optionally specify the Monitored Jobname that a monitor resourceaccepts events from. Thus, for example in the case of IMS monitor resources, youmight specify a job name of IMS1 for monitor resource MTR1 and IMS2 for MTR2.If an event arrives for OLDS and the issuer is IMS1 only MTR1 is affected.

Event TypesIn the simplest case, an event is represented by a plain message issued by a job.All monitor resources that register for a particular message accept this messageunless you also specified the monitored job name.

In other cases, for example for OMEGAMON XE situations or events reported byCICSPlex SM, the event is represented by a triggering message provided bySA z/OS for the purpose of health monitoring only. This message, ING150I, thatcontains the monitored object name or the job can then be used by SA z/OS tolocate the monitor resource and to set the health status or issue commands. Thisallows SA z/OS to handle a variety of monitoring events.

INGMON, the command that is responsible for health monitoring, is invoked fromthe NetView® automation table whenever ING150I or any other message a monitorresource has registered for is issued. It locates the monitor resource for a givenmonitored object or job and then looks up the code match table for the healthstatus or commands, or both, that should be issued whenever the triggering eventoccurs.

Code Matching for Event-Triggering MessagesINGMON allows you to pass up to three codes that, when specified, are used todetermine a specific set of commands to be issued in case of an event-triggeringmessage.

For message ING150I, SA z/OS creates an automation table entry where Code 1 isused to select commands by event severity. For other messages, you can overridethe default automation table entry and pass the appropriate tokens in Code 1,Code 2, and Code 3, as you require.

In any case, the Value Returned field contains one or two tokens separated by ablank. The first token is a required command selector that can be one of thefollowing:

selectionExecute commands with the given selection or commands for which noselection is specified.

# Perform pass processing and execute all commands that match the currentpass.

Health Monitoring


#selectionInterpret selection as another pseudo message ID. Perform pass processingfor this message and execute all commands that match the current pass.

This is useful for pass processing on behalf of the event triggeringmessage, for example, ING150I. Suppose you have one entry forWARNING and one for CRITICAL. When you do pass processing forING150I your pass counter may be on 5, for example, when the firstCRITICAL event comes in (because you already had 4 WARNING events).

However, with #selection you can specify, for example, a value returned of#MYWARN WARNING and #MYCRIT CRITICAL for the correspondinglevels. INGMON performs pass processing for the pseudo-messageMYWARN and set the health status WARNING for a WARNING event.For a CRITICAL event it performs independent pass processing for thepseudo-message MYCRIT and finally sets a health status of CRITICAL.

Remember to set the IGNORE action for the pseudo-messages to avoid ATentries being built.

The second token in the Value Returned column of the Code Processing panelindicates the optional health status to be set. If specified, it must be separated by ablank from the selection criterion.

Programming TechniquesCommands that are called by INGMON have access to the message that triggeredthe invocation using the NetView SAFE, AOFMSAFE, for example:/* MYCLIST, called by INGMON */’PIPE SAFE AOFMSAFE | STEM MSG.’If msg.0 > 0 Then

msgtext = msg.1 /* first message line */

In addition, INGMON fills the task global variables &EHKVAR0, &EHKVAR1–9,and &EHKVART with tokens that are derived from the message or exception thatINGMON was invoked by. For messages, the assignment starts with the messageID, and for exceptions, it starts with the exception ID.

INGMON also sets the following task global variables:

&SUBSAPPLContains the monitor name.

&SUBSTYPEContains the string MONITOR.

&SUBSDESCContains the description of the monitor resource.

The following examples illustrate how message and exception tokens are assignedto these task global variables.

Example 1:$HASP9211 JES MAIN TASK NOT RUNNING. DURATION- hh:mm:ss.xx


&EHKVAR0 $HASP9211

&EHKVAR1 JES

Health Monitoring



&EHKVAR2 MAIN

&EHKVAR3 TASK

&EHKVAR4 NOT

&EHKVAR5 RUNNING.

&EHKVAR6 DURATION-

&EHKVAR7 hh:mm:ss.xx

&EHKVAR8&EHKVAR9&EHKVART

NULL

Example 2:ING080I CI2XREP/MTR/KEYA OMSY4MVS OMIIMVS XREP Number of Outstanding Replies = 4


&EHKVAR0 XREP

&EHKVAR1 Number

&EHKVAR2 of

&EHKVAR3 Outstanding

&EHKVAR4 Replies

&EHKVAR5 =

&EHKVAR6 4

&EHKVAR7&EHKVAR8&EHKVAR9&EHKVART

NULL

When defining commands to be issued by the INGMON command, the&EHKVARx variables can be used to be replaced by the corresponding tokens ofthe message or exception.

When INGMON looks up the monitor resource for a given monitored object or jobname, or both, it is possible to skip monitor resource processing dynamicallythrough a user-specified REXX expression. In the absence of such a REXXexpression, INGMON locates the monitor resource with the given monitored objectname for the job that issued the message and proceeds with health status settingand commands as defined in the automation policy. By adding a REXX expressionto the User Defined Data panel within the MESSAGES/USER DATA policy itemfor the automated message, further processing can be disabled depending on theresult of this REXX expression.

To do this, the predefined keyword INGMON_FUNCTION is specified as akeyword and an arbitrary REXX expression is defined as the value in the UserData Processing panel. If the result of the REXX expression is false (that is, 0),processing is stopped, otherwise INGMON processing continues. The followingexample for the message ID MYMTR controls monitor resource processing, basedon the day of week that is defined in the common global variable DAY_OF_WEEK.(processing continues only if the current day is not a Sunday):

Health Monitoring


Predefined Keyword Data

INGMON_FUNCTION cglobal(’DAY_OF_WEEK’) \= ’SUN’

When a monitor resource is defined with a monitor command but without aninterval, the initial health status of such a passive monitor resource is obtained atmonitor resource start time only. Any other health status update must be derivedfrom events that the monitor resource has registered for.

It is however possible to issue the monitor command at any point in time byexecuting the command AOFRCMTR. This command expects the monitored objectname and optionally a job name as parameters. It locates the correspondingmonitor resource and, if specified, issues the monitor command.

See IBM Tivoli System Automation for z/OS Programmer's Reference for the syntax ofAOFRCMTR.

Health Monitoring using OMEGAMONSA z/OS allows you to interact with IBM Tivoli OMEGAMON II and IBM TivoliOMEGAMON XE products to collect key performance indicators that represent thehealth status of address spaces, middleware, or even the system.

The following sections show you how to interact with these products usingmonitor resources.

OverviewThe SA z/OS OMEGAMON interface lets you gather a wide range of performancedata on a system. You can gather data from the following performance monitoringproducts:v IBM Tivoli OMEGAMON II for MVSv IBM Tivoli OMEGAMON II for CICSv IBM Tivoli OMEGAMON II for IMSv IBM Tivoli OMEGAMON II for DB2v IBM Tivoli OMEGAMON XE productsv Other IBM Tivoli Monitoring products running on z/OS

Exception analysis is an OMEGAMON feature that monitors predefined thresholds ina system. Each time exception analysis is invoked, an exception is displayed on theOMEGAMON console if a threshold is exceeded. Using SA z/OS, you can then acton these exception alerts by running execs or issuing commands, including issuingcommands back to the host OMEGAMON.

Situations are much like exceptions but they are based on a combination of logicalexpressions and even on the status of other embedded situations. Each productbased on the IBM Tivoli Monitoring infrastructure, such as IBM TivoliOMEGAMON XE, provides a set of predefined situations that you can use as is, ormodify as you wish. You can also create your own situations to tailor themonitoring to your specific needs. Situations are edited and displayed on the TivoliEnterprise Portal (TEP). Using a TEP function called Reflex Automation, you caninform SA z/OS about a particular situation and then act upon it.

Health Monitoring


IBM Tivoli Monitoring services also allow you to interact with each and everyproduct based on this infrastructure through a standardized SOAP servicesinterface on the Tivoli Enterprise Monitoring Server (TEMS). SOAP services exist,for example, to obtain data from a particular object collected by TivoliOMEGAMON XE for z/OS. Other services allow you to automatically managesituations and TEP workflow policies, or to send universal messages to theuniversal message console.

You can set up monitor resources to:v Monitor sets of exceptions that may be of interest using an active monitor

resource and set an application's health status based on the existence of suchexceptions

v React to and resolve conditions that cause those exceptionsv Monitor sets of situations that may be of interest using a passive monitor

resource, set an application's health status and react to and resolve conditionsthat cause those situations

AssumptionsVarious topologies are possible for SA z/OS with IBM Tivoli OMEGAMON IImonitors and IBM Tivoli Monitoring products such as OMEGAMON XE:v There can be one or more monitoring product per systemv Connectivity is through VTAM® and the NetView Terminal Access Facility (TAF)

for OMEGAMON II and through TCP/IP for OMEGAMON XEv A TEMS SOAP Server is running locally, on a remote system or on a distributed

systemv SA z/OS can act as a focal point either:

– Globally, monitoring data from monitoring products running on differentsystems

– Locally, monitoring data from monitoring products running on the localsystem

The following assumptions are made about the topologies that can be adopted forinteraction with OMEGAMON II:1. The OMEGAMON product is installed on each system where MVS and CICS,

DB2, or IMS is installed.2. OMEGAMON monitors are installed and configured already to support

multiple VTAM-based connections to it. For interoperability with SA z/OS,logical units of type 3270 model 2 (24x80) are required.

3. OMEGAMON monitors are setup to interact with an external security productsuch as IBM SecureWay Security Server for z/OS (formerly RACF®).

4. OMEGAMON exceptions are reported when the threshold that is defined inOMEGAMON is exceeded. That threshold must be agreed within aninstallation because it must cater for the least severe condition that there mightbe an alert for.

The following assumption is made regarding the interaction with OMEGAMONXE:1. Reflex automation is executed on the OMEGAMON XE agent that created the

corresponding situation event

Health Monitoring using OMEGAMON


OMEGAMON InteractionThe following subsections assume that, for OMEGAMON II interaction, you havedefined one or more OMEGAMON sessions and automated functions that aredesignated to handle network communication using the SA z/OS customizationdialog.

For details on defining OMEGAMON sessions, refer to the OMEGAMONSESSIONS and AUTHENTICATION policy items in the Network (NTW) entry typeand to the OPERATORS policy in the Auto Operators (AOP) entry type describedin IBM Tivoli System Automation for z/OS Defining Automation Policy.

For OMEGAMON XE interaction using SOAP services you have to specify eachSOAP server in the automation policy that you want to connect to. For details ondefining SOAP servers, refer to the SOAP SERVER policy item in the Network(NTW) entry type described in IBM Tivoli System Automation for z/OS DefiningAutomation Policy.

Using the INGOMX Programming InterfaceINGOMX acts as the interface between operators (or auto-operators) andOMEGAMON. This includes not only any of the classic OMEGAMON monitorsfor CICS, DB2, IMS, and MVS, but also OMEGAMON XE monitors and other IBMTivoli Monitoring products running on z/OS.

For the classic OMEGAMON monitors, INGOMX can be used to issueOMEGAMON major, minor, and immediate commands, and to filter one or moreexceptions of interest from the list of exceptions reported by OMEGAMONexception analysis. Each request is written to the console (but not exposed toNetView) in the format as produced by the OMEGAMON monitor. Whenexception filtering is requested, multiple exception lines for one exception arecombined into a single line and written to the console as a single message if thefilter criterion (XTYPE) matches. INGOMX is best used within a NetView PIPE.

The INGOMX SOAP interface allows you to issue any of the SOAP servicessupported by the TEMS SOAP server, for example tov Obtain attributes of interest from a particular OMEGAMON XE object, for

example, Job_name and CPU_percent from the OMEGAMON XE for z/OSobject Address_Space_CPU_Utilization

v Start and stop situations as well as TEP workflow policiesv Issue a universal messagev Send an event into the IBM Tivoli Monitoring platform

The full set of SOAP services and a description of the XML-syntax is described inIBM Tivoli Monitoring Administrator's Guide.

The following examples illustrate the use of INGOMX. They are based on anOMEGAMON for MVS session with the name OMSY4MVS. The same techniquesalso apply to other OMEGAMON monitors. For more details, refer to IBM TivoliSystem Automation for z/OS Programmer's Reference.

Example 1. Returning Information on Common Storage Utilization Using theCSAA Command:



INGOMX EXECUTE,NAME=OMSY4MVS,CMD=CSAA| IPXNG CSAA SUMMARY| IPXNG +| IPXNG + System| IPXNG + Maximum Pre-CSAA Orphan Usage| IPXNG + ------- -------- ------- ---------------0___2___4___6___8___100| IPXNG + CSA 3312K 1247K 0 1247K 37.6%|------> || IPXNG + ECSA 307740K 78797K 0 78797K 25.6%|----> || IPXNG + SQA 1620K 660K 0 660K 40.8%|-------> || IPXNG + ESQA 145696K 23930K 0 23930K 16.4%|--> |

Example 2. Using OMEGAMON Command Modifiers:

INGOMX EXECUTE,NAME=OMSY4MVS,CMD=ALLJ,MOD=#| IPXNG #ALLJ 166INGOMX EXECUTE,NAME=OMSY4MVS,CMD=ALLJ,MOD=<| IPXNG <ALLJ *MASTER* PCAUTH RASP TRACE DUMPSRV XCFAS GRS SMSPDSE+| IPXNG + CONSOLE WLM ANTMAIN ANTAS000 OMVS IEFSCHAS JESXCF ALLOCAS+| IPXNG ...

Example 3. Trapping Outstanding Operator Replies:

INGOMX TRAP,NAME=OMSY4MVS,XTYPE=(XREP)| IPXNG + XREP Number of Outstanding Replies = 5

Example 4. Issuing OMEGAMON Minor Commands:

/* REXX-Routine EXMINOR */cmd.1 = "CMD=SYS" /* Major command, issued ahead of its minors */cmd.2 = "CMD=FCSA" /* Minor: CSA frames below 16M */cmd.3 = "CMD=FCOM" /* Minor: CSA, LPA, SQA, and nucleus below 16M */cmd.0 = 3’PIPE STEM cmd. COLLECT’,’| NETV INGOMX EXECUTE,NAME=OMSY4MVS,CMD=*’,’| CONSOLE ONLY’* IPXNG EXMINOR| IPXNG SYS >> WLM Goal mode OPT=00 SYSRES=(150526,8812) <<| IPXNG fcsa 328 1312 K| IPXNG fcom 849 3396 K

There is no need to explicitly establish a session between an operator and aparticular OMEGAMON monitor before using INGOMX; such sessions areestablished automatically on their first use.

Selective protection of individual OMEGAMON sessions and commands, or both,is possible based on the NetView Command Authorization Table. Details can befound in the appendix, “Security and Authorization”, in IBM Tivoli SystemAutomation for z/OS Planning and Installation.

To use a SOAP service, for example to obtain certain attributes from anOMEGAMON XE object, you first have to describe the request's parameters in theform of an XML document. The XML document is validated and rejected by theSOAP server if it is found to be incorrect or incomplete. The spelling of the namesenclosed in '<' and '>' is significant because XML is a case-sensitive documentdescription language. Also, because the structure of every XML document ishierarchical, each element must be enclosed by an opening name (for example,



'<CT_Get>') and a corresponding closing name denoted by a forward slashpreceding the name (for example, '</CT_Get>').

The following is an example that describes the request parameters to retrieve theJob_Name, the address space ID (ASID), and the CPU_Percent attributes from theOMEGAMON XE for z/OS object, Address_Space_CPU_Utilization, for all jobswith a CPU percentage greater than 1.0. In this example, the object that has beenqueried is collected on the TEMS called KEYAS:CMS.<CT_Get>

<target>KEYAS:CMS</target><object>Address_Space_CPU_Utilization</object><attribute>Job_Name</attribute><attribute>ASID</attribute><attribute>CPU_Percent</attribute><afilter>CPU_Percent;GT;10</afilter>

</CT_Get>

You can pass this XML document either by pointing INGOMX to a sequential orpartitioned data set, or in the default SAFE, assuming INGOMX is invoked in aNetView PIPE.

When INGOMX is invoked, the SOAP server that is connected to must bespecified. In the following example, it is assumed that you have defined a SOAPserver called KEYAYA in the SOAP SERVER policy item of the Network (NTW)entry type using the SA z/OS customization dialog. This definition includes thehost name or IP address, the SOAP server's port and the path name of the SOAPservice. The request parameters as shown above are located in the memberGETCPU in the partitioned data set SYS1.SOAP.DATA:soapds = ’SYS1.SOAP.DATA(GETCPU)’soapsrv = ’KEYAYA’Address NETVASIS ’PIPE (END % NAME GETCPU)’,’| NETV (MOE) INGOMX SOAPREQ SERVER=’soapsrv’ DATA=’soapds,’| L: LOC 1.8 ’d||’DWO369I ’||d,’| EDIT SKIPTO ’d||’RETURN CODE’||d,’ UPTO ’d||’.’||d,’ WORD 3 1’,’| VAR omx_rc’,’%L:’,’| CON ONLY’

On the successful return of INGOMX, the output of the SOAP server is returned inthe multiline ING160I message:ING160I RESPONSE FROM SOAP SERVER: 9.xxx.xxx.xxx:1920///cms/soapJob_Name:ASID:CPU_PercentIXGLOGR:20:2.1NET:59:2.1RMFGAT:89:6.9SDM1IRLM:108:1.7BBOS001S:113:22.1YANAMSJH:117:3.9

The first row of this message documents the IP address of the SOAP server thatresponded, that is, KEYAYA in the example (IP address anonymized).

The second row describes the names of the attributes returned by the SOAP server.The attribute names are separated from each other by the non-printable characterX'FF' (represented by a :).



The third and all following rows contain the actual data that has been requested.The attribute values are presented in the same sequence as the correspondingattribute names in the second row. Also, like the attribute names, the attributevalues are separated from each other by the non-printable character X'FF'(represented by a :).

The tabular structure of this message allows you to easily process it in a NetViewPIPE.

Using the INGMTRAP Monitor CommandINGMTRAP is a customized interface to INGOMX that provides filteringcapabilities for exceptions of interest as reported by OMEGAMON exceptionanalysis and triggering of automation on behalf of such exceptions. For eachexception that matches the XTYPE filter that is provided by the caller, INGMTRAPissues message ING080I, which is exposed to NetView.

For example:ING080I CI2XREP/MTR/KEYA OMSY4MVS OMIIMVS XREP Number of Outstanding Replies = 4

If no exception matches the XTYPE filter that is provided by the caller,INGMTRAP creates a ING081I message that is not exposed to NetView but writtento the monitor resource's log to document that no exception has been found. Forexample:ING081I CI2XREP/MTR/KEYA OMSY4MVS OMIIMVS NO EXCEPTION FOUND

INGMTRAP can only be used as a monitor command. This means that it has to bespecified directly as a monitor command in the definition of a monitor resource, orit has to be called on behalf of such a monitor command. The following exampleillustrates what you need to specify on the MONITOR INFO policy in entry typemonitor resource (MTR) in order to trap outstanding operator replies that arereported by OMEGAMON for MVS session OMSY4MVS:INGMTRAP NAME=OMSY4MVS,XTYPE=(XREP)

Be careful when specifying a list of exceptions: each exception may cause anING080I message to be issued. Because each occurrence of an ING080I messagetriggers health status processing of the monitor resource, make sure youunderstand the impact that this may have on the monitor resource's final healthstatus.

For more details about INGMTRAP refer to IBM Tivoli System Automation for z/OSProgrammer's Reference. For more details about defining monitor resources, refer toIBM Tivoli System Automation for z/OS Defining Automation Policy.

Health Monitoring Based on OMEGAMON ExceptionsThis section describes how to set up the monitor resources for health-basedmonitoring based on OMEGAMON exceptions using the customization dialogs,provides a sample scenario, and gives recommendations when usingOMEGAMON in combination with monitor resources.

Defining the Monitor ResourcesBy combining monitor resources and the OMEGAMON interaction methodsdescribed in “OMEGAMON Interaction” on page 52, automation can be triggeredas a result of analyzing the output reported by OMEGAMON and by the setting ofan appropriate health status.



OMEGAMON exceptions can be periodically monitored using a monitor resourceand the monitor command INGMTRAP. There are a variety of ways to handle suchexceptions:1. In the customization dialog, the MESSAGES/USER DATA policy of a given

monitor resource needs to state the health status of each exception thatINGMTRAP has been set up to monitor. Unlike messages, OMEGAMONexceptions are denoted by a '+' sign, followed by a blank and then a 4-characterOMEGAMON exception ID.

2. In addition to the health status, a series of one or more commands can bespecified to handle that particular exception. Commands are processed in thesame way as for any other resources that a MESSAGES/USER DATA policy isprovided for, such as applications (APL). This includes escalation processingbased on a PASS count, or processing based on a selection value that can bedefined using CODEs that are derived from a message.

3. The HEALTHSTATE policy can be used to issue recovery commands on behalfof an OMEGAMON exception each time the health status changes.

No matter which method or combination of method are chosen, the process ofhandling an exception is triggered by the occurrence of an ING080I message for aparticular monitor resource and exception. The automation table that is built fromthe definitions in the MESSAGES/USER DATA policy contains statements thatinvoke the INGMON command to set the monitor resource's health status and toissue commands in response to exceptions. In most cases, the necessary entries inthe NetView Automation Table are created automatically by SA z/OS. In somerare cases when, for example, command selection should be based on CODEs, it isnecessary to override the automation table definition of the exception, and tospecify up to 3 codes (CODE1, CODE2, and CODE3) on the invocation ofINGMON.

Alternatively, an installation-written monitor command can be used to issueINGOMX for a series of exceptions to one or more OMEGAMON monitor. Such amonitor command then returns with an appropriate health status that is based onthe analysis of the output produced by INGOMX. The recovery commands that areissued when the health status changes are specified in the HEALTHSTATE policyof that monitor resource.

Example ScenarioTo illustrate how SA z/OS and OMEGAMON operate together, consider thefollowing scenario.

Suppose there is a DB2 application that should be continuously monitored. Ofparticular interest is the availability of primary active logs. The LOGNexception indicates that fewer primary active logs exist than specified by therespective threshold value. This is considered a critical health indicatorbecause it can cause a DB2 hang situation if the last primary active logbecomes 100% full. Such a situation can only be resolved by making one ormore additional primary active logs available again.

In order to monitor this situation and react accordingly, the automation policy hasto be changed. First, define the session attributes for the OMEGAMON for DB2monitor, if they do not yet exist, to be able to establish a VTAM connection. TheOMEGAMON session is referred to by its session name. Then review the number ofsession operators (automation operators) that are started to handle the VTAM



session traffic and add an additional one if a higher degree of parallelism isrequired. You need to ensure that the number of session operators and predefinedNetView tasks are identical.

Next, add a new monitor resource (MTR) that periodically requests exceptioninformation from this OMEGAMON session. Add the MTR by means of aHasParent relationship to the DB2 subsystem to be monitored. This ensures that theMTR is activated when the DB2 subsystem is started, and deactivated when theDB2 subsystem is stopped. Also define the MTR via a HasMonitor relationship tothe DB2 subsystem to ensure that the monitor's health status can be propagated tothe application.

While the MTR is active, it uses the monitor command, INGMTRAP, to gatherOMEGAMON exceptions that currently exist, based on the thresholds that aredefined in the OMEGAMON for DB2 installation profile. INGMTRAP analyses allexceptions returned by OMEGAMON and filters out those exceptions that theMTR is interested in, in this example, LOGN. SA z/OS subsequently issuesmessage ING080I to initiate exception processing.

Finally, also add a new rule to the NetView automation table (using the SA z/OSpolicy) that executes a REXX automation procedure to add a new log data set tothe pool of primary active data sets whenever the LOGN exception is reported andthe health status is CRITICAL (6). The MTR's health status is considered CRITICALif the number of available primary active logs is equal to 1. If the LOGN exceptionis reported again in the next monitor interval, a second rule in the automationtable sets the MTR's health status to FATAL (7), which triggers an application movebecause normal recovery handling doesn't seem to work anymore. In addition, analert is sent to the operator to inform him about this situation. If the LOGNexception is no longer reported, the MTR's health status is set to NORMAL (3).

The health status assigned to the MTR by means of the automation table ispropagated to the DB2 application that owns this MTR. Thus, you can see at aglance whether the DB2 subsystem is okay or not.

RecommendationsYou should consider the following recommendations when using OMEGAMON incombination with monitor resources:v Avoid monitoring multiple exceptions using INGMTRAP. Note that there can be

more than one exception that may trip and thus multiple ING080I messages maybe generated. The monitor resource's health status, however, depends on the lastING080I message.

v Avoid setting different health statuses for the same exception that is monitoredby different monitor resources using INGMTRAP. Note that only one automationtable entry is generated by SA z/OS to process message ING080I for such anexception.

In these cases, the use of INGOMX, invoked from an installation-written monitorcommand, to determine a combined health status from multiple exceptions or todetermine an individual health status for each monitor resource, is preferred tousing INGMTRAP.

Health Monitoring Based on OMEGAMON XE SituationsThis section gives an overview of passive, event-based monitoring ofOMEGAMON XE situations and describes how to set up the monitor resourcesusing the customization dialogs.



OverviewUnlike the exception-based monitoring that SA z/OS uses for classicOMEGAMON monitors, the IBM Tivoli Monitoring infrastructure provides themeans to react to situations whenever they occur. On the Tivoli Enterprise Portal(TEP), a user can specify what kind of automated response (reflex automation)should be triggered for each individual situation.

SA z/OS makes use of this capability by providing a simple command calledINGSIT. The ITM administrator enters this command on the TEP with the SituationEditor dialog for those situations where SA z/OS health monitoring orhealth-based automation should take place. For more details about INGSIT refer toIBM Tivoli System Automation for z/OS Programmer's Reference.

The Take Action command is carried out on the agent, for example, OMEGAMONXE for z/OS, and not the Tivoli Enterprise Monitoring Server (TEMS) unless theTEMS is running on the same system. This is because it is possible that the hubTEMS may not reside on z/OS and so the command may not be delivered.

INGSIT triggers message ING150I that allows you to set the health status ofindividual monitor resources. It is then possible to issue commands, such asrecovery or notification commands, to automatically fix the situation. You canspecify what the health status is and what associated commands are issued in thecustomization dialog.

Defining the Monitor Resources for OMEGAMON XE SituationsTo set up the monitor resources for OMEGAMON XE Situations:1. Define one MTR for each OMEGAMON XE situation that you want to respond

to.2. In the MONITOR INFO policy item fill in the following fields:

Monitored ObjectEnter the name of the OMEGAMON XE situation in uppercase with aprefix of ITM, for example, ITM.MYSIT

Monitored JobnameEnter an optional job name to match this situation to a particularmonitor resource.

3. Define codes for the message ID ING150I in the MESSAGE/USER DATA policyof the MTR to yield the commands that are to be issued and to map theseverity to a valid health status.

Example Scenario: Consider the following scenario:

The PAGEADD command is to be issued when an auxiliary storage shortageis detected, based on page data set utilization and page data sets that are notoperational.

A situation called MyAuxShortage_Warn is defined by the installation that is truewhen both predefined situations OS390_Local_PageDS_PctFull_Warn andOS390_PageDSNotOperational_Warn are true.

As reflex automation, the following system command is issued on the managedsystem, that is, the system that produced the situations:F NETV,INGSIT MyAuxShortage_Warn,warn

Where NETV is the job name of NetView address space.



This command is issued from the Take Action dialog, as shown in Figure 7.

INGSIT is called and produces an ING150I message, which contains the situationname that is mapped to the monitored object. Other optional information includes:v The severity of the situationv A job name that matches this situation to a particular monitor resourcev Other data that contains information related to the event

In this example, the situation, MyAuxShortage_Warn, and its severity, warn, areincluded.

Using the customization dialog, a monitor resource, for example, AUXSHORT, iscreated that specifies ITM.MYAUXSHORTAGE_WARN (in uppercase) as itsmonitored object.

ING150I is then specified in the MESSAGE/USER DATA policy item of theAUXSHORT monitor resource. In this example, the following code entry could beused to derive selection ADD and set the health status to MINOR:

Code Values Entry

Code 1 warn

Code 2 *

Figure 7. Take Action Dialog



Code Values Entry

Code 3 *

Value Returned ADD MINOR

In addition, one or more commands can be specified for ING150I for the selectionthat resulted from code match processing. In the example above, the PAGEADDcommand would be specified for selection ADD.

After executing all the commands that have been specified in the CommandProcessing panel for the selection, the health status that was mapped in the codeprocessing is set (in this example, it was MINOR). Note that if no health status wasspecified in the code match table, it remains unchanged.

In a more sophisticated extension of this scenario, the situation,MyAuxShortage_Warn, as shown on the TEP is automatically acknowledged usingSOAP services. To do this, a small request parameter XML-document must becreated and sent to the TEMS SOAP server for processing. To acknowledge asituation, a CT_Acknowledge request must be issued as shown in the followingexample:<CT_Acknowledge>

<target>KEYAS:CMS</target><name>MyAuxShortage_Warn</name><source>KEYAPLEX:SYS1:MVSSYS</source><data>System Automation is taking care of this</data>

</CT_Acknowledge>

The XML-document above references the TEMS that manages the situation (target),the situation itself (name), and the so-called monitoring agent (source) that is thesource of this situation. With the data-element, you can pass any additional textualinformation to the person that is looking into this situation on the TEP.

As described in “OMEGAMON Interaction” on page 52, INGOMX is used to issuethe SOAP request to the TEMS SOAP server. Once the situation has beenacknowledged, it can be recognized as such on the TEP's situation event console ornavigator flyover list.

Health Monitoring using CICSPlex SMThis section introduces the components of event-based CICS monitoring anddescribes how to set up the monitor resources using the customization dialogs.

Component OverviewEvent-based CICS link and health monitoring is implemented using CICSPlexSystem Manager (CICSPlex SM) objects. Whenever an event is received fromCICSPlex SM, message ING150I is issued.

INGCPSM is the event listener for CICSPlex SM. Because it is a long-runningautomation procedure it needs to be run in a virtual operator station task (VOST).It scans the configuration on startup and listens for events. It then periodicallychecks whether the configuration has changed (that is, monitor resources havebeen added, deleted, or changed, etc.) or monitor resources are waiting for initialmonitoring (that is, they have STATUS=ACTIVE and HEALTH=UNKNOWN).



Creating an Application to Manage the VOSTYou can manage the VOST that executes INGCPSM using an application of typeNONMVS:v Start the VOST by using the INGVSTRT command as the start command of the

APL, where its job name is used as the attach_name of the VOST.v Stop the VOST using a sequence of INGVSTOP stop commands in the

management APL.v Monitor the status of the VOST using the INGVMON monitoring routine in the

management APL.

For more details, see IBM Tivoli System Automation for z/OS Programmer's Reference.

Defining the Monitor ResourcesTo set up the monitor resources:1. Define one MTR for each CPSM object (for example, each connection).2. Fill in the Monitored Object field in the MONITOR INFO policy item

according to the naming conventions, for example,CPSM.CICS1TOR.CONNECT.CON1

3. Leave the Monitored Jobname field empty.4. Define codes for the message ID ING150I in the MESSAGE/USER DATA policy

of the MTR to map the CPSM severities to valid health statuses, for example:

Code 1 Value Returned

VLS * NORMAL

LS * WARNING

LW * WARNING

HW * MINOR

Refer to the *CICS add-on policy for sample definitions to monitor the connectionbetween two CICS resources.

Monitoring JES3 ComponentsThe concept of a monitor resource is used to monitor the health of various JES3components. SA z/OS provides two commands that support a strict separation ofthe monitoring part and the resulting recovery processing:v AOFRJ3MN: used to monitor components in the JES3 environment, for example

spool space.v AOFRJ3RC: used to perform recovery actions against the monitored JES3 object.

The following example defines a spool space monitor:1. Define a monitor resource with a “HasParent” relationship to the corresponding

JES3 because it only makes sense to monitor the spool space when JES3 isactive.

2. Activate and deactivate commands in the MONITOR INFO policy are notnecessary for the spool monitor.

3. Use the AOFRJ3MN command as the monitor command in the MONITORINFO policy and setup the monitoring interval as desired. In this example,spool usage of up to 60% is NORMAL, 61-70% WARNING, 71-80% MINOR,81-90% CRITICAL and greater than 90% FATAL.

Health Monitoring using CICSPlex SM


||

||||

AOFRJ3MN JES3_subys SPOOLSHORT 60,70,80,90

4. Define the recovery action in the HEALTHSTATE policy, for example:

State Command Text

NORMAL AOFRJ3RC JES3_subsys SPOOLSHORT RESET

CRITICAL AOFRJ3RC JES3_subsys SPOOLSHORT 05

FATAL AOFRJ3RC JES3_subsys SPOOLSHORT 01

Issue one recovery command every minute. The commands are read from theJES3 SPOOLSHORT CMDS policy of the JES3 subsystem. When the spool usagegoes down to 60% or less, the health status goes to NORMAL. This causes theinvocation of the AOFRJ3RC command but now with the RESET option - theRESET option stops recovery. It is recommended that you use JESOPER as theauto-operator for the recovery commands. Note, that the recovery commandsfor the SPOOLSHORT condition must be defined for the JES3 subsystem.

5. For the JES3 subsystem, define the necessary actions that should be performedfor SPOOLSHORT in the JES3 SPOOLSHORT CMDS policy:

Pass Automated Function Command

1 JESOPER MVS &SUBSCMDPFXF U,Q=HOLD,AGE=30D,N=ALL,C




This purges all jobs from the hold queue that are older than 30 days in the firstpass. On pass 2, all jobs older than 10 days are purged. On pass 3 all jobs olderthan 3 days are purged. Finally, after 10 times the pass interval (in our example5 minutes), all jobs older than 1 day are deleted if the recovery action is notreset in the meantime.

AOFRJ3MN RoutineUse this routine to monitor various objects in a JES3 environment.

The following objects can be monitored:v MDS queues (Fetch queue, Verify queue, Wait volume queue, Error queue,

Allocation queue, Breakdown queue, Unavailable queue, Restart queue, Systemselect queue, System verify queue)

v Current setup depthv Spool space

For each of the 10 JES3 MDS queues, thresholds may be set for each of the 4 healthstatuses (Warning, Minor, Critical and Fatal) indicating the number of jobs thatparticular queue may contain causing to set the corresponding health status. If, forexample, the WARNING threshold for the Error queue is set to 5, if 5 or more jobsare pending on the MDS Error queue, the health status is set to Warning.

For the spool space the thresholds define the amount of used space that whenexceeded causes to set the corresponding health status.

Whenever AOFRJ3MN is called, it issues the appropriate JES3 command (*I,Q,S forSPOOLSHORT and *I,S for the MDS queues) and parses the response. The valueextracted from the message text is compared with the thresholds and then the

Monitoring JES3 Components


|||

|||

||

||

|||||

||

||||

|||

|||

|||

|||||||||

return code is set to the corresponding health status. This simply sets the healthstatus of the Monitor resource (MTR). No recovery action is taken by AOFRJ3MNroutine. Use the HEALTHSTATE policy of the Monitor resource to define arecovery action for each health status, if necessary.

The syntax of the AOFRJ3MN routine is as follows:

►► AOFRJ3MN jes3apl object threshold-list ►◄

object:

MDSCOUNTQMDSCOUNTFMDSCOUNTVMDSCOUNTWMDSCOUNTEMDSCOUNTAMDSCOUNTBMDSCOUNTUMDSCOUNTRMDSCOUNTSSMDSCOUNTSVSPOOLSHORT

threshold-list:

warning,minor,critical,fatal

jes3aplSpecifies the name of an APL of category JES3 for which this monitorworks.

monitorSpecifies the JES3 object to be monitored:

MDSCOUNTQCurrent setup depth

MDSCOUNTFFetch queue

MDSCOUNTVVerify queue

MDSCOUNTWWait volume queue

MDSCOUNTEError queue

MDSCOUNTAAllocation queue

MDSCOUNTBBreakdown queue

MDSCOUNTUUnavailable queue



MDSCOUNTRrestart queue

MDSCOUNTSSSystem select queue

MDSCOUNTSVSystem verify queue

SPOOLSHORTSpool

threshold-listSpecifies a list of four threshold values separated by commas:

warningSet health status to WARNING if this value is exceeded

minor Set health status to MINOR if this value is exceeded

criticalSet health status to CRITICAL if this value is exceeded

fatal Set health status to FATAL if this value is exceeded

If warning is not exceeded the health status is set to NORMAL.

Note that for SPOOLSHORT the values are in percent but for the MDSqueues they are absolute numbers. No value checking is done byAOFRJ3MN except for whole numbers.

Note also that the thresholds are tested from FATAL to WARNING. So ifyou want to go directly from NORMAL to FATAL, you could specify50,50,50,50

AOFRJ3RC RoutineThis routine performs the recovery action against a monitored object in a JES3environment.

When AOFRJ3RC is called, it checks whether the system that it is running thatholds the JES3 global processor. If not AOFRJ3RC terminates without any furtheraction.

The syntax of the AOFRJ3RC routine is as follows:

►► AOFRJ3RC jes3apl msg-type pass-intervalRESET

►◄

jes3apl Specifies the name of an APL of category JES3.

msg-typeSpecifies the message type within the given JES3 APL that the recoverycommands are to be read from:

pass-intervalSpecifies the time interval that AOFRJ3RC should wait beforeexecuting the next pass. The format is in NetView notation ( mm,hh:mm, hh:mm:ss or :ss).

RESETIf RESET is specified AOFRJ3RC stops the recovery.



AOFRJ3RC looks into the MESSAGE/USER DATA policy definition of the specifiedJES3 APL. It issues the command that is defined for PASS1 of the given messagetype. As long as there are commands in higher passes it sets up a NetView timerthat re-calls AOFRJ3RC after the given pass interval. Whenever AOFRJ3RC isexecuted the command that is defined for the next pass is issued as long as oneexists.

If RESET is specified instead of a pass interval any pending timer is killed andprocessing stops.

The return code is always zero.

Note: AOFRJ3RC issues the recovery commands in a fire-and-forget manner. It doesnot check whether the recovery action has the desired result. This is done by themonitor. After one or more monitor intervals the health status changes to a lesssevere one if the recovery shows an effect. If you want to stop recovery actionswhen the health status returns to NORMAL, for example, you have to code aHEALTHSTATE command that calls AOFRJ3RC with RESET.

JES2 Spool MonitoringAn SA z/OS monitor resource (MTR) is used to monitor JES2 spool file usage.This can be accomplished with an active monitor that queries the spool usageperiodically or a passive monitor that listens for HASP050 and HASP355 events.

The JES2 spool monitoring function that is provided includes the following items:v Automation routines INGRMJSP, INGRCJSP (AOFRSD01), AOFRSD09, and

AOFRSD0H. See “INGRMJSP” on page 219, “INGRCJSP (AOFRSD01)” on page220, “AOFRSD09” on page 212, and “AOFRSD0H” on page 216.

v Automation table entries for system messages HASP050 and HASP355.v Configuration parameters for the JES2 spool recovery process in the JES2

SPOOLSHORT and JES2 SPOOLFULL policy items of the JES2 application.

DB2 Connection MonitoringSA z/OS allows you to monitor DB2 connections for both CICS and IMS:

CICS The CICS command CEMT INQUIRE DB2CONN is issued regularly aftereach monitor interval to query the status of the CICS DB2 connection.

For more details, see the sections “Monitoring of CICS DB2 Connections”and “INGRMCDB Routine for the Monitoring of CICS DB2 Connections”in IBM Tivoli System Automation for z/OS Product Automation Programmer'sReference and Operator's Guide

IMS The IMS command DISPLAY SUBSYS is issued regularly after eachmonitor interval and the response to this command is analyzed withrespect to the status of the connection to a DB2 subsystem.

For more details, see the sections “Monitoring of IMS DB2 Connections”and “INGRMIDB Routine for the Monitoring of IMS DB2 Connections” inIBM Tivoli System Automation for z/OS Product Automation Programmer'sReference and Operator's Guide



IMS Component MonitoringFor IMS automation, SA z/OS enables the monitoring of online log data sets(OLDS) and recovery control data sets (RECON) of IMS control regions, and allowsthe status checking of the VTAM Application Control Blocks (ACB) and theenablement of logons.

The monitor routines that are provided for this and the necessary definitions toenable the monitoring functions are described in IBM Tivoli System Automation forz/OS Product Automation Programmer's Reference and Operator's Guide.

IMS Component Monitoring


Chapter 5. Joblog Monitoring

This information introduces the components of joblog monitoring and describeshow to set up the monitoring function using the customization dialogs.

OverviewJoblog monitoring is designed to monitor JES spool files only. The currentimplementation supports any JES spool output file regardless of whether the job isexecuting or the job has finished. However, in the latter case the output must stillbe available on the JES output queue and the user has to ensure that the output isprocessed only once.

To use joblog monitoring, NetView must have a JES job ID. If the user enables theDSIRQJOB task, the JES job ID can be obtained by starting the DSIRQJOB. If theuser disables the DSIRQJOB task, the INGTJLM task will automatically acquire aJES job ID as needed when joblog monitoring is started. Before any attemptedshutdowns of JES in both cases, the user must ensure that the job ID is released bystopping either the DSIRQJOB or INGTJLM task, whichever they have employed(refer to the best practice policies for SA's recommended approach). Alternatively,refer to Starting the NetView Program Before Starting JES topic in the NetViewInstallation: Getting Started manual. This topic explains how to use the NetViewMVS Command Revision function to stop the DSIRQJOB job when JES endsabnormally or is stopped by a user from the command line.

The monitoring function is controlled by SA z/OS based on the definitions of thecustomization dialog. Monitoring is automatically started when the job reaches astatus of UP, ACTIVE or RUNNING. If your job was already active at the time youspecified the joblog monitoring definitions in the customization dialogs, then thisjob must be restarted in order to start joblog monitoring for it. However, you canmonitor jobs which are not controlled by SA z/OS with the following restrictions:1. You cannot specify any filter criteria to limit the data that is passed to

automation. This means any line of data except an empty line which isgenerally excluded is forwarded to the message automation.

2. The message that is forwarded to message automation is always queued to theautotask LOGOPER. For SA z/OS controlled jobs those messages are queuedto the autotask that is responsible for the job in view of SA z/OS.

3. You need to start the monitoring manually using the command INGJLM START.Required parameters are the job name and the monitoring interval. In case youwant to monitor a data set other than the default data set JESMSGLG you alsoneed to specify the appropriate ddname. The specification of the owner or thejob id are necessary only when multiple jobs with the same job name exist. Thiscould be the case when the job already ended and the output is held on theoutput queue. In this case it is your responsibility that the job is monitoredonly once.

4. You need to qualify the ddname by the step name if you do not want tomonitor a spool file of the last step in a multi-step job when the ddname isspecified multiple times. In case the step name is not unique in an in-streamprocedure you need to qualify the ddname by the procedure step name as well.

You can stop the monitoring function for a particular job at any time using thecommand INGJLM STOP even for SA z/OS controlled jobs. Normally, the


||||

https://www.ibm.com/support/knowledgecenter/SSZJDU_6.2.1/com.ibm.itnetviewforzos.doc_6.2.1/inq_grsnv_befjes.htm

monitoring is automatically stopped when the job has ended. For SA z/OScontrolled jobs this is done when the job reaches a termination status likeAUTOTERM, ENDING, and so on. For non-SA z/OS controlled jobs themonitoring is automatically stopped when the job has ended and the monitoringinterval has expired twice.

The monitoring task can be suspended for an indefinite time frame using thecommand INGJLM SUSPEND. The accumulated output of all monitored jobs isprocessed not before the task has been restarted. The output of jobs that havefinished in the meantime is lost unless the output is still held on the JES outputqueue. Jobs that have been started after the task was suspended will not bemonitored after the task has been restarted. To restart the task use the NetViewcommand START TASK=INGTJLM.

In case of an abend condition the monitoring task performs an internal suspendcommand with the following exception. The job that caused the error condition ismarked "in error" and will be excluded from monitoring when the task is restarted.

Note: The monitoring task must be terminated before JES is shut down. For thisreason the default policy has been updated and the following STOP command hasbeen added to the SHUTINIT phase of JES:

PIPE NETV INGJLM STATUS| LOC /INGTJLM: ACTIVE/| EDIT /STOP TASK=INGTJLM/ 1| NETV

Limitations1. Executing SA z/OS controlled jobs which are running less than two seconds

are probably not monitored. One reason is that the automation does not findthe job active any longer. Or, JES has deallocated the resource before themonitoring task could allocate it. The latter case is also true for non-SA z/OScontrolled jobs when the output is not held on the JES output queue after thejob ended. In any case specify a message class when starting the job that leavesthe output on the JES output queue and trigger the monitoring manually.

2. The monitoring function is limited to the primary subsystems JES2 and JES3.3. Dynamically allocated spool output data sets are not supported.4. When the NetView task DSIRQJOB is defined, Job Log Monitoring waits for

receiving the JES job ID by DSIRQJOB. Without the job ID, a spool data setcannot be accessed. If DSIRQJOB is not defined, the job ID is obtained by JobLog Monitoring and returned to JES when the monitoring task terminates.If DSIRQJOB is terminated in a JES2 environment, the monitoring task isautomatically suspended because the job ID is returned to JES2. And themonitoring task is automatically restarted after DSIRQJOB is restarted.In a JES3 environment, DSIRQJOB does not return the job ID on termination.For this reason, the monitoring task is not terminated. However, if for whateverreason the job ID is returned to JES3, the monitoring task is automaticallysuspended but needs to be restarted manually when a new job ID is received.

CustomizationFor SA z/OS controlled jobs you have to define the monitoring interval, themessages and data sets to be monitored for each job in the customization dialog.The main definition which actually initiates the monitoring for a job is thedefinition of the monitoring interval. You specify the interval at the policy"APPLICATION INFO".


|||||||

|||

||||

|

Note: Without the interval definition no monitoring takes place regardless of whatthe policy "MESSAGES/USER DATA" specifies.

Dedicated messages that should be automated must be defined by specifying theoffset of the message id within a single line and optionally one or more tokensmaking the message unique. An offset value of 0 indicates a message without aparticular id. In this case at least one token pair must be defined to identify themessage. In case all messages are relevant and should be automated the commonmessage id value JOBLOGALL is to be specified. This id does not require any furtheruser data specification. However, the dialog will not generate an ACF fragmentunless the message id has attached any of the definitions "S", "C", "R", "K", or "U".For this reason, the user data keyword/value pair JLM_OFFSET=NO is required.

When specific messages should be monitored, the keyword JLM_OFFSET must definethe actual offset of the message ID in the corresponding SYSOUT data set. For theJESMSGLG data set, the offset of all text begins at a specific column:

JES2 environmentThe text begins at column 20 regardless of whether it is a multi-line message orthe message text is wrapped because it is longer than 126 characters (includingthe 19-byte prefix). Follow-on lines of a multi-line message show the console IDof the multi-line message at column 4. Wrapped message text shows blanks inthe first 19 columns. Thus, trapping tokens of such messages requires theknowledge of the exact position of each token. The first two tokens of the firstline of each message are the time and the job ID.JES3 environment

Application Information Line 00000001Command ===> ________________________________________________ Scroll ===> CSR__

Entry Type : Application PolicyDB Name : SAMPLE_PDBEntry Name : JLMS05 Enterprise Name : KEYPLEX:JCL Procedure Name . . . . AAAZJLMS____________________Job Log Monitor Interval . 00:17 (mm:ss NONE)Captured Messages Limit . ___ (0 to 999):

Figure 8. ISPF dialog defining the joblog monitoring for an application. The dialog accepts aninterval time from 1 second up to 1 hour.

Message Processing Line 00000001 Col 001 075Command ===> ________________________________________________ Scroll ===> CSR__

Entry Type : Application PolicyDB Name : SAMPLE_PDBEntry Name : JLMS05 Enterprise Name : KEYPLEX:Cmd Message id Description Cmd Rep Cod Usr A Mu__ JOBLOGALL Automate all JESMSGLG messages_________ * *:

Figure 9. ISPF dialog defining the joblog monitoring of all JESMSGLG messages (1/3)

User Data Processing : JOBLOGALL Line 00000001 Col 001 075Command ===> ________________________________________________ Scroll ===> CSR__:Cmd Keyword Data___ JLM_OFFSET__________ NO____________________________________________________:


Chapter 5. Joblog Monitoring 69

The text begins at column 12 regardless of whether it is a multi-line message orthe message text is wrapped because it is longer than 126 characters (includingthe 11-byte prefix). The first token of each message is always the time.

Other SYSOUT data sets certainly have different layouts.

The first token definition in the sample below ends with the character '*'. Thisindicates that the message tokens are checked for all characters up to the starcharacter only. The star character and all immediately following non-blankcharacters are ignored. The indicated message is passed to automation when its idis found at column 21 and the next token after the id has the value '3' and the 7thtoken begins with KEY=.

In case you want to monitor data sets other than the JESMSGLG data set you needto qualify your message definition(s). The following sample shows you that theSYSOUT data sets referenced by the ddname AAAZOUT and SYSPRINT passevery message to automation.

The next example shows you the monitoring definitions of a multi-step job. Allmessages of the job steps STEP1 and STEP3 that are written to the ddnameSYSPRINT are passed to automation. But, messages of STEP2 written to SYSPRINTare ignored.

The following example shows you how to define a filter criteria that is notassociated with a message. You still need to define a "dummy" message id as ananchor. All lines of the sysout data set defined by ddname AAAZOUT whose first

User Data Processing : cccnnnna Line 00000001 Col 001 075Command ===> ________________________________________________ Scroll ===> CSR__:Cmd Keyword Data___ JLM_OFFSET__________ 21_______________________________________________________ JLM_TOKEN___________ (7,’KEY=*’)______________________________________________ JLM_TOKEN___________ (4,’3’)_______________________________________________:

Figure 11. ISPF dialog defining the automation of specific messages. The value of the tokenmust be enclosed in apostrophes.

User Data Processing : JOBLOGALL Line 00000001 Col 001 075Command ===> ________________________________________________ Scroll ===> CSR__:Cmd Keyword Data___ JLM_OFFSET__________ NO_______________________________________________________ JLM_DDNAME__________ SYSPRINT_________________________________________________ JLM_DDNAME__________ AAAZOUT_______________________________________________:

Figure 12. ISPF dialog defining the joblog monitoring of specific messages. The parameterJLM_DDNAME defaults to JESMSGLG when omitted.

User Data Processing : JOBLOGALL Line 00000001 Col 001 075Command ===> ________________________________________________ Scroll ===> CSR__:Cmd Keyword Data___ JLM_OFFSET__________ NO_______________________________________________________ JLM_DDNAME__________ STEP1.SYSPRINT___________________________________________ JLM_DDNAME__________ STEP3.SYSPRINT________________________________________:

Figure 13. ISPF dialog defining the joblog monitoring of specific messages of a multi-step job


||||||||||||

||||

|||

token has the value ’Name’ are passed to automation.

The build process will automatically generate the following common MAT entryfor automating the message INGY1300I but only for SA z/OS controlled jobs:

This just needs the definition of the command(s) that should be executed on behalfof the message id, for example:

Note that the message INGY1300I is a multi-line message. The first line is the labelline which provides the following tokens OWNER, JOBNAME, JOBID, DDNAME,MSGID in the sequence as shown. The latter token is exactly the message id thatyou have specified on the 'Message Processing' dialog. Each subsequent linerepresents a line of the original message.


Entry Type : Application PolicyDB Name : SAMPLE_PDBEntry Name : JLMS05 Enterprise Name : KEYPLEX:Cmd Message id Description Cmd Rep Cod Usr A Mu__ DUMMY1 Automate specific text line____________ * *:

Figure 14. ISPF dialog defining the joblog monitoring of a non-message print line (1/2)

User Data Processing : DUMMY1 Line 00000001 Col 001 075Command ===> ________________________________________________ Scroll ===> CSR__:Cmd Keyword Data___ JLM_OFFSET__________ 0___________________________________________________________ JLM_DDNAME__________ AAAZOUT_____________________________________________________ JLM_TOKEN___________ (1,’Name’)____________________________________________:

Figure 15. ISPF dialog defining the joblog monitoring of a non-message print line (2/2)

* INGY1300I for jobs defining a message idIF MSGID = ’INGY1300I’ & TOKEN(6) ¬= ’N/A’& TOKEN(3) = SVJOB & TOKEN(6) = MSGTYPE THEN

DISPLAY(N) NETLOG(Y) SYSLOG(N)EXEC(CMD(’ISSUEACT JOBNAME=’SVJOB’ MSGTYPE=’MSGTYPE)

ROUTE(ONE %AOFOPGSSOPER%));

Figure 16. Common MAT entry for message INGY1300I and jobs defining a message id formonitoring. The value N/A indicates a message coming from a job that has not defined amessage id for monitoring.


Entry Type : Application PolicyDB Name : SAMPLE_PDBEntry Name : JLMS05 Enterprise Name : KEYPLEX:Cmd Message id Description Cmd Rep Cod Usr A Mc__ JOBLOGALL Automate all JESMSGLG messages_________ 1 * *:



|

If you want to monitor jobs that are not defined in the policy or that are defined inthe policy but do not have a message id defined for monitoring you need tomanually adjust the existing, predefined AT entry (create an AT override) like thefollowing:

In both cases you have to start the monitoring manually. And, since no filtercriteria are defined all messages are passed to automation.

Note: When you use the NetView PIPE stage SAFE for retrieving the message youwill find the message in the default safe.

SPIN Parameter

For a spool data set:JESMSGLGJESYSMSGSYSOUT ddname (JES2 only)

that is defined to be spun off and to be monitored by Joblog Monitoring, it ishighly recommended to hold the output until Joblog Monitoring has processed thedata set. Otherwise, you may lose information you would like to automate.

For the JESMSGLG data set you also need to verify that the output is notsuppressed. In a JES2 environment the command$D JOBCLASS(STC),LOG

must show the value YES for the parameter LOG. In a JES3 environment thecommand*I,STD

must now show the value NOSTC for the parameter JESMSG. And, the command*I,C=x

must not show the value NOLOG for the parameter JESMSG where 'x' is the classthat is assigned to the started task jobs.

When Joblog Monitoring detects that a particular data set has been spun off by JESit issues the message INGY1333I after it has finished its processing of the data set.The message informs you that the particular spool data can be released now. Forexample, the job name or the job identifier along with the SYSOUT identifier of themessage can be used to purge the data set:JES2: $P O JQ(jnm!jid),OUTGRP=soidJES3: *F,U,J=jnm!jid,DSN=...soid,CANCEL

Note that 'jid' is just the numerical part of the job identifier.

* INGY1300I for jobs having no message id definedIF MSGID = ’INGY1300I’ & TOKEN(6) = ’N/A’ THEN

DISPLAY(N) NETLOG(Y) SYSLOG(N)EXEC(CMD(’xxxxxxxx ...)

ROUTE(ONE %AOFOPGSSOPER%));

Figure 18. Common MAT entry for message INGY1300I and jobs without defining a messageid for monitoring


||||||

|||||

|||||

||

||

If the SYSOUT identifier is not present in the message the task could not determinethe value.

The message INGY1306I indicates that monitoring has been stopped for aparticular ddname and all corresponding spool data sets can be released.

Status InformationThe command INGJLM STATUS which can also be issued in a NetView PIPE returnsthe current status of the monitoring task and its monitored data sets:

The status of a data set shows the following information:E........

The data set is marked "in error" and is not monitored any longer..A.......

The data set is allocated...O......

The data set has been opened successfully....U.....

The user is not allowed to read the data set.....S....

The monitoring of the data set is ACF-based......M...

All messages are passed to automation.......J..

The corresponding job has not been found on the chain of active addressspaces.

.......I.The spool data set is not initialized. The allocation of the data set is retriedat the next monitoring interval. This repeats until the data set has beeninitialized.

........NThe ddname has defined the SPIN attribute. That means the correspondingspool data set can be spun off during processing.

The frequency shows the interval in minutes and seconds between two checks foraccumulated messages. The next column shows the time when the last checkoccurred. The last two numbers show the total number of messages that have beenread as well as the number of messages that have been passed to the automationtable. The second status line of each data set shows the number of message readand passed to automation since the last check.

Status of task INGTJLM: {ACTIVE|INACTIVE|SUSPENDED}Owner Jobname Jobid DDname Status Freq. Last Read Passed

Procstep Step LRead LPassed-------- -------- -------- -------- --------- ----- -------- ------- -------xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxx mm:ss hh:mm:ss n n

n nxxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxx mm:ss hh:mm:ss n n

xxxxxxxx xxxxxxxx n nxxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxxx mm:ss hh:mm:ss n n

xxxxxxxx n n*** Status complete ***

Figure 19. Joblog monitoring status information


Chapter 6. Alert-Based Notification

SA z/OS provides an alert-based notification service that enables you to alertsubject-matter experts. You can escalate automation problems that require manualintervention by sending alerts, events or trouble tickets to different kinds ofnotification targets.

SA z/OS supports several communication methods that allow you to deliver alertsto notification targets such as:v System Automation for Integrated Operations Management (SA IOM)v Tivoli Enterprise Console (TEC)v Tivoli NETCOOL/OMNIbusv IBM Tivoli Service Request Manager®

v A user-defined alert handler

OverviewThe alert-based notification service of SA z/OS allows alerts to be sent tooperators or system programmers for predefined situations.

You can also customize when to issue alerts, if desired, using the customizationdialog and the INGALERT utility. Alerts can only be issued for applications (APL),monitor resources (MTR), application groups (APG), and MVS components.

An alert is a set of information that is collected and sent by an SA z/OSautomation agent to a target for notification processing. The information that issent consists of the text that is to be forwarded to the alerted person or group. Thisinformation is supplemented by additional options that determine in detail theprocessing at the different kinds of notification targets.

In SA z/OS there are several predefined alert points that trigger alerts whenever acommand encounters a problem situation, such as a resource becoming degradedor not being up within a given time interval.

Alerting can be enabled or disabled at various levels:v Globally using the INGCNTL commandv Resource-specific using the resource's Inform Listv Alert-specific using code definitions for the message ID INGALERT

Communication FlowFigure 20 on page 76 outlines the communication between the automation managerand the automation agents.


The automation agents on the systems in the sysplex subscribe to the automationmanager to be alerted about problems with system or sysplex application groups(APGs). This is because the automation manager cannot itself send alerts to thenotification targets. Whenever the automation manager detects a problem with anAPG it sends an alert to the subscribed automation agents (one in case of a systemAPG, and all in case of a sysplex APG). Any alerts for sysplex APGs are handledby only one automation agent in the sysplex.

The automation agents can also receive alerts for applications, application groups,or monitor resources via the INGALERT command. If the affected resource ismanaged by a different automation agent, the request is passed on. Theautomation agent that manages the resource sends the alert to the notificationtarget. If, for whatever reason, this automation agent cannot send the alert, itpasses on the request to the next automation agent in the sysplex. This can happenseveral times until the alert is successfully sent or no more automation agents areavailable.

For each alert, the automation agent connects to a notification target, sends thealert and then disconnects. The automation agent does not maintain a permanentconnection to the SA IOM server.

Enabling AlertingBy default alerting is not enabled. To activate it you must perform setup actions inboth SA z/OS and notification target.

Setup in SA z/OSYou can turn alerting on or off at three different levels in SA z/OS:v The system level, via the INGCNTL command. Turning off alerting means that

no alerts are detected or accepted by the system. Alerting must be turned onexplicitly either globally or selectively for at least one notification target.

v The resource level, via the Inform List policy field. Turning off alerting meansthat no alerts are detected or accepted for the resource. The notification targetmust be explicitly specified (or inherited from the defaults) to activate alertingfor it.

Sysplex

Automation Manager

Detects problem with APG,and notifies all subcsribed agents

SYS1

Automation Agent

Detects problem withAPL or MTR on SYS1

Handles alerts for systemand sysplex APGs

SYS2

Automation Agent

Detects problem withAPL or MTR on SYS2

Handles alerts for systemand sysplex APGs

pass request

subscribefor APGs

subscribefor APGs

INGALERT

Alert for APL, APG, or MTRon SYS1 or SYS2

notify agent

notify agent

Figure 20. Alert Communication Flow

Alert-Based Notification


v The alert ID level, via code definitions for the INGALERT message ID of theresource or MVS component entry.

INGCNTL CommandBy default alerting is not enabled. You have to issue the INGCNTL command toenable it and set the connection properties for the notification target.

This can be done as follows:v In the NetView style sheet using auxiliary commands:

*********************************** Auxiliary commands*********************************** Enable Alerting and set connection propertiesauxInitCmd.A = INGCNTL SET ALERTMODE=IOM ALERTHOST=saiom:1040:SAALERT

v From the AOFEXDEF exit that is supplied with SA z/OS:’INGCNTL SET ALERTMODE=IOM ALERTHOST=saiom:1040:SAALERT’

See IBM Tivoli System Automation for z/OS Programmer's Reference for moreinformation about the INGCNTL command.

Inform ListYou have to include the appropriate communication method for the notificationtarget in the Inform List field of the appropriate policy item to explicitly enablealerting for specific resources or classes of resources, as shown in Table 7.

Table 7. Inform List Policy Items

Policy Object Policy Item

Application Group (APG) APPLGROUP INFO *

Application (APL) APPLICATION INFO *

Monitor Resource (MTR) MONITOR INFO *

MVSCOMP Defaults (MDF) MVSESA INFO *

System Defaults (SDF) AUTOMATION OPTIONS

Sysplex Defaults (XDF) RESOURCE INFO* Leaving the Inform List field blank allows the policy object to inherit the value specifiedin the system defaults or sysplex defaults definition.

You must also specify the appropriate communication method for the desirednotification target as shown in Table 8.

Table 8. Inform List Communication Methods

Value in Inform List Communication Method Supported NotificationTarget in SA z/OS

IOM Peer-to-peer protocol of IBMTivoli System Automation forIntegrated OperationsManagement (SA IOM)

SA IOM

EIF Tivoli Event IntegrationFacility (EIF)

TEC or OMNIbus

TTT XML TSRM via TDI

USR Command call User-defined alert handler


Chapter 6. Alert-Based Notification 77

You can specify a blank-separated list of values to enable alerting for severalnotification targets.

Code ProcessingCode processing with the INGALERT message ID allows you to define additionalcharacteristics for events to be passed to the notification target or to prevent eventcreation for certain alerts. Such definition can be made, dependent on the alert ID,issuing job and type of notification target.

Code definitions for message ID INGALERT can be used for resources of typeAPL, APG, MTR, and for MVS components. If no matching code definitions arefound for the APL, APG or MTR resources, the INGALERT code definitions arechecked for the corresponding MVS component entry on the system where theresource resides.

Enter the following in the Code Processing panel for the INGALERT message ID:

Code 1The alert ID that identifies the type of alert. SA z/OS provides thefollowing set of built-in alert points:

Alert ID Description For Resource Type

CMD_FAILED Return code checking is on and thecommand ended with RC≠0

APL, MTR

COMM_FAILED An error was detected duringcommunication to another system

APL

CRITICAL_WTOR WTOR defined as CRITICAL withoutdefined REPLY.

APL

ISQHWMSG Important Hardware Messages areavailable on the target hardware.

MVSESA

ISQHWST Target hardware is in DEGRADED orSERVICE REQUIRED status.

MVSESA

MSG/Message ID Messages with a critical severity. Themessage can be abbreviated by means ofwildcard, for example, MSG/DFS54*.

This Alert ID is applicable only formessages that are captured by theAOFCPMSG command.

APL, APG, MTR

OS_DEGRADED The observed status of the resource hasbecome degraded

APL

OS_PROBLEM The automation status is ZOMBIE orBROKEN, or a shutdown outsideSA z/OS and restart is not allowed

APL

REC_FAILED Automation was halted because the criticalthreshold for a minor resource wasexceeded

APL

REP_FAILED No further outstanding WTORs that arestored by SA z/OS need to be replied to

APL

START_FAILED The start command failed APL

START_PENDING The up message was not received withinthe timeout interval

APL

STOP_PENDING Ran out of stop commands APL



||||

Alert ID Description For Resource Type

CS_PROBLEM The compound status PROBLEM has beenset

APG

OPCERR Indicates a status change to a TWSresource as reported by message EVJ120I

APL

Note: Alert points ISQHWMSG and ISQHWST are provided by ProcessorOperations for target hardware connected over SNMP protocol.

You can also use any user-defined alert ID. Simply specify it in thecorresponding code entry and call INGALERT with this ID. Wildcards aresupported.

Code 2For APL this is the job name that alerting should be done for. For MVC itcontains MVSESA. For APG and MTR Code 2 is ignored. Wildcards aresupported. This allows you to set alerting for several APLs at once byusing APL classes.

Code 3The communication method that is used to send the alert to thenotification target. Valid values are IOM, EIF, TTT or USR. Wildcards aresupported.

Value ReturnedThis can be either IGNORE to prevent event creation, or parameters that aresent to the notification target together with the passed event. The meaningof these parameters depends on the type of communication method, asfollows:

IOM The first two tokens of the Value Returned are considered to be:v The priority of the alert (0–999).v The escalation ID that is used in SA IOM to define the rules that

determine how the alert should be processed. The length of thisvalue is limited to 20 characters.

If the first two tokens have invalid values, the Value Returned isassumed to be IGNORE.

If you specify more than two tokens in the Value Returned field,the superfluous tokens are ignored.

EIF The Value Returned is considered to be the event severity. Validvalues are HARMLESS, WARNING, MINOR, CRITICAL or FATAL,or a corresponding number between 1 and 5, where 1 correspondsto HARMLESS, etc. Both alternatives for specifying a severity canbe used for events to TEC or NETCOOL/OMNIbus. Whenspecifying the severity as a number, the code definition can also beused to send alerts to SA IOM.

If you do not specify a valid severity, the Value Returned isassumed to be IGNORE.

Superfluous tokens in the Value Returned field are ignored.

TTT If TSRM is the notification target, the values in Value Returned areused as:v The priority of the trouble ticket as it is initially reported (1-5)



||

v The urgency, which is a indication of how quickly a troubleticket should be resolved (1-5)

v The business impact or severity of the trouble ticket (1-5)

These values are not validated because other targets may expectother values.

If you specify more than three tokens in the Value Returned field,the superfluous tokens are ignored. If you specify less than threetokens, they are used according to their position and the missingtokens default to N/A.

USR The content of the Value Returned field is passed to theuser-defined alert handler that is called.

Code Definitions Example:Consider the example in Table 9.

Table 9. Code Processing Example for the INGALERT Message ID

Code 1 Code 2 Code 3 Value Returned

START_FAILED IMS* IOM 500 IMS_start

START_FAILED DB2* EIF CRITICAL

* * * IGNORE

The code definitions in this example result in the following behavior:v Alerts with the alert ID START_FAILED for jobs with the name prefix IMS are

sent to IOM with priority 500 and escalation ID IMS_start.v Alerts with the alert ID START_FAILED for jobs with the name prefix DB2 are

sent as EIF events to TEC or NETCOOL/OMNIbus with event severityCRITICAL.

v All other alerts are ignored for all notification targets.

INGALERT CommandYou can use the INGALERT command to inject alerts into a system. This can befrom either the NetView automation table, an automation procedure, or thecommand line.

You can specify the following parameters:v A resource name , the text MVSESA, or a job or subsystem name.v The alert ID, for example, CS_PROBLEM, CMD_FAILED, and so on.v A message ID that identifies the message text or a text string that is passed to

the notification target.

For example, the following can be used from the command line or an automationprocedure:INGALERT MYGRP/APG/SYS1 ID=MYALERT TEXT=(MYGRP HAS A PROBLEM)

In this example, INGALERT uses the alert ID, MYALERT, to obtain additionalparameters via a matching code definition for the message ID INGALERT, and ituses the TEXT parameter value for the alert text.

The following can be used from the NetView automation table to send an alertwhenever message ABC123I is issued:



IF MSGID='ABC123I'THENEXEC(CMD('INGALERT'));

INGALERT uses ABC123I as the alert ID and the complete text of messageABC123I as the alert text. The resource parameter of INGALERT is defaulted to thejob name of the subsystem that issued the message.

See IBM Tivoli System Automation for z/OS Programmer's Reference for moreinformation about the INGALERT utility.



Chapter 7. Availability and Recovery Time Reporting

SA z/OS introduces support to assist you in billing users or reporting reliability ofyour critical applications or the software that those applications are dependent on.For example, you might want to charge accurately based on the amount of timerequired to run an application. This is of importance for non-MVS resources, suchas USS applications, or monitoring resources that might run in the NetViewaddress space.

OverviewSA z/OS collects and records job-related information, and writes SystemManagement Facility (SMF) records at specific events in the lifetime of a resource.

This resource can be:v A subsystem (APL)v An application group (APG) that is hosted by the local system as well as sysplex

application groupsv A monitor resource (MTR)

The INGPUSMF batch utility produces a report file that you can import into aspreadsheet. You can also convert and write the report into DB2 tables that areprovided and exploited by the IBM Tivoli System Automation ApplicationManager. For more details, see “Writing the SMF Report to DB2” on page 88.

You can control whether a record is written for a resource by entering the valueSMF in the Inform List field in the resource's information policy item.

Resource LifecycleFigure 21 shows the events in the lifetime of an application when SA z/OS recordsinformation.

These events are:

t0 t1 t2 t3 t =t4 0

Last downphase

Startupphase

Activephase

Shutdownphase

Sta

rtin

g

Active, running

Stop

ping

Start order orexternal start

Stop order orexternal stop

UP event DOWN event

Figure 21. Events in the Lifecycle of an Application


v Start order received from the automation manager (t1)v UP signal received (t2)v Stop order received from the automation manager (t3)v DOWN signal received (t4=t0)

By examining these records you can establish the following information for a giventime period:v Application up time and downtimev Application startup and shutdown timev The number of scheduled stoppages and the approximate amount of scheduled

downtimev The number of unscheduled stoppages and the approximate amount of

unscheduled downtime

To make using SA z/OS SMF records easier, the following periods areautomatically calculated and stored (in units of seconds) in the SMF record:v The startup time (t2–t1)v The shutdown time (t4–t3)v The time the application was active (t3–t2)v The last down time (t1–t4)

You therefore have a precise view of the lifecycle of the application.

Layout of the SMF RecordTable 10 provides details of the data that is stored in the SMF record.

Table 10. Layout of the SMF Record

Offset Length Format Description

00 2 Binary Record length. This field and the next (a total of 4 bytes)form the record descriptor word (RDW).

02 2 Binary Segment descriptor. This is zero.

04 1 Binary System Indicator Bit:0 Reserved1 Subtypes used

05 1 Binary SMF Record Type. This is 114.

06 4 Binary The time, since midnight, that the record was moved intothe SMF buffer (in hundredths of a second).

10 4 Packed The date when the record was moved into the SMFbuffer, in the form 0cyydddF.

14 4 EBCDIC System Identification (from the SID parameter).

18 2 Binary Record subtype:1 Automation tracking record

20 2 Binary Record version.

22 2 — Reserved.

24 4 Binary Offset to product section from start of record, includingthe record descriptor word (RDW).

28 2 Binary Length of product section.

30 2 Binary Number of product sections. This is always 1.

Availability and Recovery Time Reporting


Table 10. Layout of the SMF Record (continued)

Offset Length Format Description

32 4 Binary Offset to resource section from start of record, includingthe record descriptor word (RDW).

36 2 Binary Length of resource event section.

38 2 Binary Number of resource event sections. This is always 1.

40 8 — Reserved.

Product Section

00 16 EBCDIC Product name, for example SA z/OS V3R2M0.

16 8 EBCDIC Name of the SYSPLEX.

24 8 EBCDIC Domain identifier.

32 8 EBCDIC MVS System name.

40 8 EBCDIC XCF group name.

Automation Section

00 24 EBCDIC Resource name (in automation manager notation).

24 8 EBCDIC Job name (optional).

32 2 Binary Event type:X'0001'

StartingX'0002'

ActiveX'0003'

StoppingX'0004'

InactiveX'0005'

Degraded

34 2 — Reserved.

36 12 EBCDIC Automation agent status (optional).

48 12 EBCDIC Start type.

60 12 EBCDIC Stop type.

72 5 EBCDIC Termination type (abend code). Optional.

77 3 — Reserved.

80 4 Binary Total startup time in seconds.

84 4 Binary Elapsed time in seconds that the resource was active.

88 4 Binary Total shutdown time in seconds.

92 4 Binary Last down time of resource in seconds.

Enabling SMF RecordsTo enable SMF records for a resource:1. Ensure that the SMFPRM xx member in SYS1.PARMLIB is set up to collect type

114 SMF records by adding type 114 to the SYS(TYPE statement:SYS(TYPE(30,...,114)

2. Specify SMF in the Inform List of the APPLICATION INFO policy item for theresource.


Chapter 7. Availability and Recovery Time Reporting 85

The INGPUSMF UtilityYou can use the INGPUSMF utility to analyze SMF records and produce a data setthat can be imported into a spreadsheet program. The data set contains the type114 records that SA z/OS produces in a format that can easily be imported. Bydefault, the fields are semicolon delimited.

OutputThe first record in the data set is a title record that describes each column. Theremaining records are the data records. One data record is written for each type114 SMF record.

Table 11 describes the format of each record.

Table 11. Format of INGPUSMF Utility Data Set Records

Column Description

1 SMF system ID

2 Date when SMF record was written, in YYYYMMDD format

3 Time when SMF record was written, in hhmmss format

4 SA z/OS product name, including release level

5 Name of sysplex

6 System name

7 NetView domain ID

8 XCF group name

9 Resource name in automation manager notation

10 Job name, if present

11 Event

12 Automation agent status

13 Startup time in seconds

14 Active time in seconds

15 Shutdown time in seconds

16 Down time in seconds

17 Start type

18 Stop type

19 Termination (abend) code

The INGPUSMF Utility JCLThe INGPUSMF utility runs as a batch job. See INGEUSMF for a sample. Themeaning of the DD statements is as follows:

STEPLIBThe load library that contains the INGPUSMF utility. The utility resides inthe SINGMOD1 library.

REPORTThe output data set that contains the spreadsheet import data set in asemicolon-delimited format. The record size is 255 bytes.



SYSPRINTContains information that is written by the utility.

HSATRACEIs used for debugging purposes only. If present, the INGPUSMF utilitywrites trace entries to record the process flow.

SMFDATAContains the SMF records. The record format is: Variable, blocked,spanned.

USRPARMSContains user options, such as filter criteria or a specific separatorcharacter.

User OptionsYou can specify various options in the USRPARMS data set that control theprocessing of the utility. You must specify each option in a separate record. Theoption are defined as keyword=value pairs. If you specify an option several times,the last occurrence is used. The keyword must start in column 1 of the record. Noblanks are allowed in front of or after the equal sign (=). A asterisk (*) isconsidered to be a comment.

The following options are supported:

SEPCHAR=charDefines the separator character to be used to separate the columns. Thedefault is a semicolon (;) if omitted

SYSIDDefines the SMF system ID used as a filter. Only SMF records that aregenerated by that system are taken. The value can be 1–4 characters.

FROM=dateThe starting date used as a filter. The format is YYYYMMDD. All SMFrecords written on the specified date or later are taken.

TO=dateThe ending date used as a filter. The format is YYYYMMDD All SMFrecords that are written no later than the specified date are taken.

RESOURCE=Defines the resources in automation manager notation used as a filter. Youcan specify up to 10 resource names. The name can be a wildcard, such as*abc, abc* or *abc*.

Return CodesThe following return codes are set by the utility:

0 Normal completion.

8 Invalid option detected in the USRPARMS data set.

12 REPORT data set is not accessible.

16 A severe error occurred, for example, an open error for the SMFDATA data set,or writing a record to the REPORT file.



Writing the SMF Report to DB2You can convert and write the SMF report that is produced by INGPUSMF intoDB2 tables that are provided and exploited by the IBM Tivoli System AutomationApplication Manager.

The following reports are provided:v Startup and Shutdown Reports:

– Report the cumulative startup and shutdown times for a resource, includingits dependencies.

– Report resources with the longest startup and shutdown times in a selecteddomain.

v Availability and Recovery Reports:

– Report a resource's uptimes and downtimes, unexpected outages andcorresponding recovery times.

– Report resources that had the highest number of unexpected outages in aselected domain.

A conversion utility, known as the z/OS offloader, delivers the z/OS domain datathat is required to run these reports.

The z/OS offloader component runs as a batch job (see Figure 22) and usesexisting and new programs that are installed into the end-to-end automationadapter HFS directory, which is normally /ust/lpp/ing/adapter.

You can use the sample job INGXRPRT to run the z/OS offloader.

CustomizationAfter installing the z/OS offloader you must carry out the following customizationsteps:

Figure 22. SMF Processing with z/OS Offloader



1. Customize the script /usr/lpp/ing/adapter/ingreport.sh. Adapt the installationpath:INSTALL_DIR=/usr/lpp/ing/adapter

2. Copy the sample job INGXRPRT and follow the steps as described in it. Thereare several input parameters that you need to set correctly otherwise theconversion utility cannot access the DB2 table:

Parameter Details

INGDSN=HLQ.SMF.REPORT The data set of the SMF report created by theINGPUSMF utility.

INGSEPCHAR=; This must be the separator as used by the INGPUSMFutility.

INGDOMAIN=MyDomain The name of the E2E domain as specified in the E2Eadapter configuration file, ing.adapter.plugin.properties.If omitted the default is used, which consists of thesysplex name and XCF group name.

INGDB2_USER=db2inst1 The DB2 user name for remote logon.

INGDB2_PSW=db2admin The DB2 password for remote logon.

INGDB2_PORT=50000 The TCP/IP port to connect to the remote DB2.

INGDB2_SERVER=db2-host-name The TCP/IP host name to connect to the remote DB2.

INGDB2_NAME=EAUTODB The DB2 name or the DB2 location if DB2 resides onz/OS.

INGDB2_SCHEMA=EAUTOUSR The DB2 schema of the table.

3. (Optional) If the database is located on a z/OS system, a DB2 license file isrequired. An appropriate license file for z/OS platform, db2jcc_licencse_*.jarmust be installed in the application classpath. Connectivity to z/OS databasesis enabled with the license file as defined by the following table.

Update DB2 database From==> To License file required

Distributed system ==> z/OS DB2 db2jcc_license_cisuz.jar

z/OS system ==> z/OS DB2 db2jcc_license_cisuz.jar

z/OS system ==> distributedDB2

db2jcc_license_cu.jar

a. Copy the appropriate license file, for example, from DB2_INSTALL_PATH/db2/db2v8/jcc/classes/db2jcc_license_cisuz.jar to the directory/usr/lpp/ing/adapter/lib.

b. Modify the classpath in the script /usr/lpp/ing/adapter/ingreport.sh andadd the license file for example: DB2_LICENSE+$INSTALL_LIB/db2jcc_license_cu.jar

4. Run the INGXRPRT job that copies the SMF report to DB2.

OutputThe output of the ingreport.sh shell script shows the progress of the z/OSoffloader. Any errors that occur are reported in this output. See IBM Tivoli SystemAutomation for z/OS Messages and Codes for details of these messages (INGX9850E,INGX9855E, and INGX9856E).



Chapter 8. Automating Processor Operations-ControlledResources

This information describes how to customize your SA z/OS installation to enablethe automation of messages coming from target systems that are controlled byprocessor operations.

These target systems or resources are referred to as processor operations resourcesin the following.

Processor Operations, which is a focal point type function, allows you to monitorand control processor hardware including Coupling Facility images, from a singleNetView, the processor operations focal point.

Notes:

1. VM guest systems are treated the same as any other target systems that arecontrolled by ProcOps (see IBM Tivoli System Automation for z/OS Operator'sCommands for details).

2. PSMs are "virtual" hardware and therefore not all target hardware commandsapply (see IBM Tivoli System Automation for z/OS Operator's Commands fordetails).

With the method described in this chapter, you can use SA z/OS systemoperations to react on these messages. This information is contained in“Automating Processor Operations Resources of z/OS Target Systems Using ProxyDefinitions,” which introduces the general process how to achieve such messageautomation.

Automating Processor Operations Resources of z/OS Target SystemsUsing Proxy Definitions

SA z/OS processor operations can be used to automate messages that cannot beautomated on the target systems themselves. Typically these messages includethose appearing at IPL time.

In a sysplex environment there are additional messages (XCF WTORs) beingdisplayed at IPL time when joining the sysplex and at shutdown time when asystem is leaving a sysplex. These WTOR messages cannot be automated yetbecause SA z/OS system operations is not active at that time.

With the XCF message automation framework described in this chapter, you havea method of exploiting your own XCF message automation.

Note: There are XCF WTOR messages which are automatable by Sysplex FailureManagement (SFM). In these cases, to avoid conflicting automation, it is notrecommended that you automate these messages by SA z/OS.

ConceptYou can use the SA z/OS standard interface and routines to handle systemexternal messages in almost the same way as system internally generated


messages. This applies to the way of defining message automation in thecustomization dialog as well as to the means available for controlling messageautomation at automation time.

To exploit the system operations mechanism for message automation, a proxyresource representing the processor operations resources must be generated in thecustomization dialog as entry type Application (APL).

There is a one-to-one relation between a proxy and a processor operations resource(target system). How to implement this relation in the customization dialog isdescribed in the following subsections.

Messages that are generated on external systems, where no SA z/OS is active ornot yet active, can also be automated. The resources generating these messages arecalled processor operations resources. They are defined in the customization dialog asentry type System (SYS).

Customizing Automation for Proxy ResourcesIt is assumed that you have already used the customization dialog to defineprocessor operations target systems and made these systems accessible to theprocessor operations focal point via the Processor Control file (see also IBM TivoliSystem Automation for z/OS Defining Automation Policy). So for every processoroperations target system that has been defined on the processor operations focalpoint, you should define a proxy resource. You do this by defining the proxyresource as entry type Application (APL) in the customization dialog.

Note: If you want to define many proxy resource applications, you can use theapplication class concept as described in IBM Tivoli System Automation for z/OSDefining Automation Policy).

Defining the proxy resource as an Application (APL) has another advantage: Thesystem is then visible in the INGLIST panel and it can be managed and monitoredlike an application resource. SA z/OS users are able to not only use messageautomation for target system messages, they can also issue start and stopcommands to IPL and shut down systems. These commands can be defined likeany start and stop command for an application. Unlike application resources, targetsystems are managed by processor operations commands (for example, ISQCCMDtarget_system_name ACTIVATE FORCE(NO) or ISQSEND target_system_name OCvary xcf,target_system_name,off,retain=yes). Processor operations commands allowyou to send MVS commands to target systems as well as hardware commands tothe processor (support element).

The rules that you need to obey when defining the proxy resource are as follows:1. You need to define (or have defined) the processor operations target systems

that you want to automate. For those systems, the following rule applies:MVS SYSNAME = ProcOps name

The MVS SYSNAME must be identical with the ProcOps name.If this is not the case, you need to change it subsequently.

2. Job Name = ProcOps name

The Job Name of the application for the proxy resource must match theprocessor operations target system's name as defined when creating thissystem in the customization dialog.

3. Job Type = NONMVS

The Job Type for the proxy application must be NONMVS.

How to Automate Processor Operations Controlled Resources


4. The Monitor Routine for the proxy application must be ISQMTSYS.5. Sysname = MVS SYSNAME

The Sysname for the proxy application must match the MVS SYSNAMEdefined for the processor operations target system. This definition is used forresource monitoring.

6. If you want to inhibit operators from performing a startup or shutdown for atarget system resource using the INGREQ command, External Startup andExternal Shutdown must be set to 'ALWAYS'.

7. If you do not want the proxy resource to be automatically started, you shouldset the Restart after IPL option to NO.

8. Because you can only automate applications by linking them to systems via anapplication group, you need to define an application group for the proxyapplications. Do not merge the proxy applications with other applications intothis application group because destructive requests applied to a mergedapplication group would also affect the proxy resources contained in thatgroup.You may choose PASSIVE behavior to not forward requests against theapplication group to each member. This prevents you from unintentionallysending requests to processor operations target systems represented by theirproxies.

9. In the Message Processing panel for the proxy application define the messagesto be automated in the Message ID column. Do not specify message IDISQ900I, as this message is used as a carrier for the original target systemmessage.Enter cmd in the Action column to specify the command to be processed if thedefined message occurs.

10. If the message to be automated is a WTOR, the variable &EHKVAR1 containsthe reply ID. This variable may then be used as a parameter to the ISQSENDcommand:ISQSEND &SUBSJOB OC R &EHKVAR1,COUPLE=00

Startup and Shutdown ConsiderationsProcessor operations commands must be used to start or stop processor operationsresources, for example:v Start example:

ISQCCMD &SUBSJOB LOAD FORCE(NO)

v Stop example:Pass 1 ISQSEND &SUBSJOB OC Z EOD________________________________________________________________________

Pass 2 ISQSEND &SUBSJOB OC VARY XCF,&SUBSAPPL,OFF,RETAIN=YES

Note: If the delay time between sending the commands in pass 1 and pass 2 isnot appropriate, you can define a resource-specific Shut Delay in the ApplicationAutomation Definition panel.

For more details about processor operations commands refer to IBM Tivoli SystemAutomation for z/OS Operator's Commands.

Preparing Message AutomationThe interaction with target systems is based on the SA z/OS processor operationscomponent. Therefore the installation and customization of this component mustbe complete at this point.


Chapter 8. Automating Processor Operations-Controlled Resources 93

Operating System messages from processor operations target systems receiving atthe focal point are transferred to ISQ900I messages.

ISQ901I is not relevant. It is used to inform interested operators about targetsystem messages. It is not used for automation purposes.

MSCOPE() parameter in CONSOLxx member: MSCOPE allows you to specifythose systems in the sysplex from which this console is to receive messages notexplicitly routed to this console. An asterisk (*) indicates the system on which thisCONSOLE statement is defined. Because the default is *ALL, indicating thatunsolicited messages from all systems in the sysplex are to be received by thisconsole, this parameter must be set to '*' for correct automation by SA z/OSprocessor operations.

Automating Linux Console Messages

The Linux Console Connection to NetViewWhen a Linux target system IPLs, its boot messages are displayed on the ConsoleIntegration facility (CI) of the System z® or 390-CMOS processor Support Element(SE). For SA z/OS processor operations, CI is the only supported interface tocommunicate with the Linux operating system.

The communication between the processor operations focal point and CI is basedon the NetView RUNCMD and the Support Element's Operator Command Facility(OCF), an SNA application. In SA z/OS processor operations, this connection pathis referred to as a NetView Connection (NVC).

Linux Console Automation with Mixed Case Character DataUnlike operating systems which translate console command input into uppercasecharacters, Linux is case sensitive. The NetView automation table syntax allows theuse of mixed case characters in compare arguments of an IF statement.

When an automation command is to be scheduled as a result of such acomparison, any message token arguments passed, are not translated intouppercase by NetView. Make sure that your automation routine does not do anuppercase translation of parameters passed. For example, in REXX use thestatement PARSE ARG P1 P2 instead of ARG P1 P2, which implicitly performs atranslation into uppercase. If a Linux message invokes your automation code andthe message information is retrieved using NetView's GETMLINE function, nouppercase translation occurs. In order to send mixed case command data to theLinux console consider the following REXX statement:Address Netvasis ’ISQsend MYlinux Oc whoami’

The addressed REXX command environment, Netvasis, passes the command stringwithout doing an uppercase translation. The ISQSEND command internallytranslates its destination parameters into MYLINUX and OC but leaves commandwhoami as is.

Security ConsiderationsAfter Linux system initialization, usually a LOGIN prompt message is displayedallowing users defined to the system to login. The ISQSEND command interfacedoes not suppress any password data from being displayed.



You may use the NetView LOG suppression character to avoid the passwordinformation to be visible in the NetView log. In Support Element log files, suchpassword data can be viewed in text form.

Restrictions and LimitationsThe following Linux systems are supported:v Linux systems running in an LPAR of a System z or 390-CMOS processor

hardwarev Linux systems running on a System z or 390-CMOS processor hardware,

configured in Basic modev Linux systems running as VM guest machines under z/VM® Version 4.3 or

higher

Linux systems running under a VM, which itself runs as a VM guest, are notsupported.

In the command shell environments of a Linux console it is possible to pass controlkeys as character strings instead of pressing the keyboard control key combinationto perform functions like Control-C. The current Linux support of SA z/OSprocessor operations has not been tested using this Linux capability. Any Linuxprogram or command script that requires a user interaction with control keysshould not be invoked using the SA z/OS processor operations ISQSENDinterface.

How to Add a Processor Operations Message to AutomationUse the NetView automation table (AT) and the SA z/OS command set toimplement console automation. You can automate the routine functions that anoperator performs when a particular message is generated. For more informationsee IBM Tivoli System Automation for z/OS Defining Automation Policy, SC33–7039.

Messages Issued by a Processor Operations Target SystemWhen a target system issues a message, the message is forwarded to the processoroperations focal point system. The focal point system repackages the messagewithin an SA z/OS ISQ900I message, an ISQ901I message, or both, and routes themessage to the appropriate task:v ISQ900I messages are routed to SA z/OS processor operations autotasks. If you

want automation that you write to receive ISQ900I messages, use the ISQEXECcommand to run the automation in a target control task. For information aboutusing the ISQEXEC command, see section Sending an Automation Routine to aTarget Control Task in “Issuing Other OCF Commands” on page 12. YourNetView automation table entries for SA z/OS should acknowledge the ISQ900Iidentifier for all target system messages forwarded to the processor operationsfocal point system. You can specify your ISQ900I automation table entries to betarget system specific, however, this is not recommended.

v ISQ901I messages are routed to all logged-on operators identified as interestedoperators by the ISQXMON command or marked as such in the customizationdialog.

For information about the ISQEXEC and ISQXMON commands, see IBM TivoliSystem Automation for z/OS Operator's Commands.

A message forwarded from an SNMP connection consists of the following:

Automating Linux Console Messages


v ISQ900I or ISQ901I message identifierv Name of target system where the message originatedv Console designator form describing where the message originatedv Message identifier and text of the original message from the target system

For example, if a NetView connection forwards the message IEA101A SPECIFYSYSTEM PARAMETERS from the operating system to the focal point system, SA z/OScreates one or both of the following SA z/OS messages:ISQ900I target-system-name OC IEA101A SPECIFY SYSTEM PARAMETERSISQ901I target-system-name OC IEA101A SPECIFY SYSTEM PARAMETERS

This message format applies to all processor operations target system messages. Itis independent of the target system resource that generated the original message.

The processor operations target system message is sent in the same format as itwould be displayed on the processor Support Element (SE) or HardwareManagement Console (HMC).

Specifics of VM second level systems: Messages from guest machine operatingsystem appear in the following format:ISQ900I psm-name.guest-name OC IEA101A SPECIFY SYSTEM PARAMETERS

Messages from CP on the virtual machine appear in the following format:ISQ900I psm-name.guest.name OC HCPGSP2627I The virtual machine is

placed in CP mode due to a SIGP initial CPU reset from CPU 00.

Messages from the PSM itself appear in the following format:ISQ700I psm-name SC ISQCS0314E Message Handler has failed.

Note: Make sure your consoles issue messages in the format that you expect andwrite your NetView automation table entries accordingly.

Sample NetView Automation Table StatementsThe following message response example presents a request for system parameterswhen the message ID string contains 'IEA101A':IF TEXT = . ’IEA101A SPECIFY SYSTEM PARAMETERS’

& MSGID = ’ISQ900I’ .THEN EXEC( CMD(’ISQI101 ’ ) ROUTE ( ONE * ))

DISPLAY(N) NETLOG(Y);

This NetView automation table statement initiates the ISQI101 routine when themessage condition is true.

Note: Text within messages may be in mixed case. Be sure your coding accountsfor mixed case text.

Message ISQ211ISome SA z/OS commands attempt to lock and unlock ports. Where an operatorowns the lock for a port, the SA z/OS unlock command, ISQXUNL, returns RC=12associated with message ISQ211I Unable to unlock target name console.

In such a case, you have the choice of either using the ISQOVRD command toforce an unlock or you may end your automation with a message. Thereafter, youcan view your NetView log to find out the reason for the lock of the port.

How to Add a Processor Operations Message to Automation


Your automation may encounter this message ISQ211I frequently. Attempting tounlock a locked port is not an error condition; however, it may be a sign that thecalling command did not succeed. Schedule your automation from messages thatindicate positively that a command did not run, not from the ISQ211I message.

Processor Operations Command MessagesSome SA z/OS commands run on the target system. The message returned fromthese commands indicates only that the support element was told to schedule theoperation. Consequently, the operation at the target system may not complete eventhough the SA z/OS message indicates a successful completion.

SA z/OS acknowledges only that the command was successfully forwarded to thesupport element. An unsuccessful operation at the target system generates anunsolicited message that the support element forwards to the focal point system inan ISQ900I message. Schedule your automation from the message that positivelyindicates that a target system operation did or did not complete.

The SINGSAMP SA z/OS sample library contains the PL/I source code for severalautomation routines that issue responses to selected messages. You can select theresponse that is most appropriate for your enterprise. You can also use them asmodels to create your own automation routines. The list in Table 12 summarizesthese routines, the messages they respond to, and the responses they issue initially.

Table 12. SINGSAMP SA z/OS Sample Library Routines

SINGSAMPMember Routine Description

INGEI120 ISQI120 Responds to the following messages:

IEA120A Device ddd volid read, reply cont or wait.IOS120A Device ddd shared (PR volid not read.)

the recovery task, reply cont or wait.

Issues the following response to the target: CONT

INGEI357 ISQI357 Responds to the following message:

IEE357A Reply with SMF values or U.

Issues the following response to the target: U


$HASP426 Specify options - subsystem_id.

Issues the following response to the target: WARM,NOREQ.


ICH502A Specify name for primary/backupRACF data set sequence nnn or none.

Issues the following response to the target: NONE


IEA877A Specify full DASD SYS1.DUMP data setsto be emptied, tape units to be used asSYS1.DUMP data sets or GO.

Issues the following response to the target: GO



Table 12. SINGSAMP SA z/OS Sample Library Routines (continued)

SINGSAMPMember Routine Description


IEE956A Reply - ftime = hh.mm.ss,name = operator,reason = (ipl,reason)or u.

Issues the following response to the target: U

The SA z/OS automation table entries in the ISQMSG0 member of theSINGNPRM data set include inactive entries that call these automation routines. Toincorporate these routines into your automation, do the following:1. Remove the comments from the corresponding automation table entries for the

messages that initiate the automation routines you want to use. If you performthese steps as part of the initial SA z/OS installation, make these changesbefore you incorporate the SA z/OS entries. If you do this after the initialSA z/OS installation, change the NetView automation table.

2. Code the routines you are using to issue the responses you want.3. Compile the PL/I source code for the routines you want to use, and link the

resulting object code to your PL/I library.4. Recycle the NetView program to activate the new entries.

For automation processing to occur, each message in the NetView automation tableat the focal point system and at each target system must be made available to thesystem's NetView program. In z/OS, MPF controls message availability to theNetView program. Examine the MPF list member in the SYS1.PARMLIB data set toensure that the necessary messages are marked for automation. For target systemsusing other operating systems, check the message suppression facilities used onthose systems.

Testing MessagesSA z/OS provides a collection of NetView automation table entries for yourSA z/OS configuration. NetView automation table entries are in the AOFCMDmember of the SA z/OS SINGNPRM installation data set. When these entries aremoved to your NetView automation table, they may need additional editing.

For example, you may already test for a particular message in your productionNetView automation table. If you add an entry that tests for that same message,your automation table will not run as you expect. After a match with the testcriteria is found, the search of the automation table is aborted. The secondNetView automation table statement is not found. Consequently, the message doesnot drive all of your required actions.

To avoid this, combine entries into a single test condition. This ensures that allrequired actions are scheduled for all messages. For the following message:IEA320A RESPECIFY PARAMETERS OR CANCEL

your NetView automation table may already have the following entry: (▌1▐)IF MSGID = ’IEA320A’THEN EXEC (CMD(’USERJOB’) ROUTE( ONE *) ) CONINUE(Y);

With SA z/OS installed, the following message appears when forwarded fromSystem z or 390-CMOS processor hardware:



ISQ900I SYS1 OC IEA320A RESPECIFY PARAMETERS OR CANCEL

After the SA z/OS entries are added, the NetView automation table includes thefollowing entry:IF TEXT = . ’IEA320A RESPECIFY PARAMETERS’ .

& MSGID = ’ISQ900I’ .THEN

EXEC (CMD(’ISQI320 ’ ) ROUTE( ONE *) )DISPLAY(N) NETLOG(Y);

In this case, the first entry satisfies the IF test and the command USERJOB runs(▌1▐). The second command, ISQI320, is not scheduled to run because once themessage matches a table entry, the autotask stops searching. Combine these twoentries into a single entry, such as:IF TEXT = . ’IEA320A RESPECIFY PARAMETERS’ .

& MSGID = ’ISQ900I’ .THEN

EXEC(CMD(’ISQI320 ’ ) ROUTE( ONE *) )EXEC(CMD(’USERJOB ’ ) ROUTE( ONE *) )DISPLAY(N) NETLOG(Y);

When you use the second example, both commands are scheduled.

If your NetView automation table tests the text of SA z/OS messages, the messageformat must match the character case for which you test. This can be done byrequiring all sites to use the same format for their messages, or by duplicating ATentries in uppercase and in mixed formats.

Building the New Automation DefinitionsWhen you are finished using the customization dialog to add message responseand automation operator information to the automation policy, you need to buildthe system operations control files.

The complete description of how to build and distribute these files is provided inIBM Tivoli System Automation for z/OS Defining Automation Policy.

The SA z/OS build function places the new automation definitions in the data setdefined in the Build Parameters panel.

Copy the new automation definitions into the SA z/OS NetView DSIPARMconcatenation in the NetView startup procedures, or concatenate it to the NetViewDSIPARM data set.

Loading the Changed Automation EnvironmentTo reload the AMC file, automation control file and the AT perform the followingactions:

To reload the MPF list, enter the following command:v From the z/OS console:

SET MPF=xx

v From a NetView console using the MVS prefix:MVS SET MPF=xx

Where xx is the suffix of the MPF member in the SYS1.PARMLIB data set to load.



To reload the automation manager configuration file, all updated automationcontrol files and the automation tables issue:INGAMS REFRESH

Specify a data set name or an asterisk (*) which means reload the current data set.

Using Pipes and ISQCCMD for Synchronous HW CommandsThe System Automation for z/OS Hardware interfaces command, ISQCCMD,available for Processor Operations SNMP connections and with imitations for BCPInternal Interface connections, allows the management and control of processorsand logical partitions, as well as hardware activation profiles.

When used in automation procedures, ISQCCMD provides an easy-to-use interfaceto automate processor operations management and configuration tasks.

The following HW commands return all their response information immediately toNetView on command completion and are therefore called synchronous commands:v CCNTLv CONDATAv CPCDATAv GETCLUSTERv GETISTATv GETIINFOv GETSDGAv GETSINFOv GETSSTATv ICNTLv PROFILEv STPDATAv TCDATA

For SNMP and BCPii connections, ISQCCMD supports NetView PIPES. Oncompletion of the ISQCCMD command, a PIPE KEEP with the name ISQ.SNMPcontains the immediate command response of the HW command that was issued,for example:* ISQCCMD G14 GETSINFO| ISQ417I GETSINFO STATUS(SUCCESS)| ISQ900I G14.KEY3 SC AOFA0017 GETSINFO G14 STATUS(OPERATING) PDATA(TYPE(2084)

,MODEL(B16),S/N(000020016F7A)) MODE(LPAR) APROF() CPCSNAME(IBM390PS.G14) NAME(G14)TSTIME(070825131936)

| ISQ419I ISQCCMD GETSINFO processing on G14 is complete.* IPSFO PIPE KEEP ISQ.SNMP | CONS| IPSFO AOFA0017 GETSINFO G14 STATUS(OPERATING)

PDATA(TYPE(2084),MODEL(B16),S/N(000020016F7A)) MODE(LPAR) APROF()CPCSNAME(IBM390PS.G14) NAME(G14) TSTIME(070825131936)

In the example above, the HW common command GETSINFO was issued at aNetView console. Embedded in the ‘ISQ’ messages the response from the hardwareis displayed on the console, starting at report ID AOFA0017.

The same information is available if you reference the PIPE KEEP with the nameISQ.SNMP, once the ISQCCMD command completed, as shown in the example,with the content of ISQ.SNMP displayed on the console.

Loading the Changed Automation Environment


In an automation procedure, this can be coded as shown in the following example:/*ReXX*//* Display CPC information using the ISQ.SNMP KEEP */Arg cpcname'ISQCCMD 'cpcname' GETSINFO'If RC = 0 Then Do

'PIPE KEEP ISQ.SNMP ' ,' | LOC /AOFA0017/ ' ,' | LOC /'cpcname'/' ,' | CONS ONLY'

End

As an alternative, you can get the immediate ISQCCMD HW responses directlyinto the PIPE input stream if you use the PIPE NETVIEW stage followed by anEXPOSE TOTRAP stage. In this case, all ISQ messages and the AOFA0017 reportdata is available for PIPE processing./*ReXX*//* Display CPC information in a PIPE */Arg cpcname'PIPE NETV ISQCCMD 'cpcname' GETSINFO' ,' | EXPOSE TOTRAP ' ,' | LOC /ISQ90/ , /* takes ISQ901I or ISQ900I */' | LOC /AOFA0017/ ,' | LOC /'cpcname'/' ,' | CONS ONLY'

Automating Asynchronous Hardware Commands with ISQCCMD andPIPES

The following ISQCCMD hardware commands return two messages to NetView.First a message that the HW command has either been accepted for execution orrejected. Second, if an acceptance message was issued, a completion event messagethat contains the actual success or failure information of the command is sentasynchronously.v ACTIVATEv CBUv CTRLCONSv DEACTIVATEv EXTINTv LOADv OOCODv RESERVEv RESTARTv STARTv STOPv STPv SYSRESETv TCM

Automation scripts using the ISQCCMD interface must distinguish between theaccepted or rejected response of an asynchronous HW command and the actualcommand completion information, which may either indicate successful executionor a failure. The asynchronous command completion events from the hardware aremade available for message automation and TRAP AND WAIT processing by

Using Pipes and ISQCCMD for Synchronous HW Commands


ProcOps. Application scripts using the ISQCCMD interface can get the Accepted orRejected responses directly at ISQCCMD termination time. The Accepted responsecan then be used to wait for the command completion event message.

Member INGEI004 of the SINGSAMP library provides a REXX sample illustratinghow asynchronous hardware commands can be automated using ISQCCMD andNetView PIPES, together with TRAP and WAIT.

VM Second Level Systems SupportThis feature provides ProcOps support to control and monitor guest machinesrunning under VM.

ProcOps allows an operating system to be IPLed into a processor, amongst otherfacilities. Such an operating system is VM. Within VM other operating systems canbe IPLed as guest machines. Of particular interest are LINUX guest machines, butMVS, VSE and even VM guest machines may be possible. (Lower levels of guestmachines are not considered). Previously there was no effective way to entercommands to and receive messages from such a guest target system in order tovalidate that it had IPLed correctly, or that it is behaving correctly.

With second level guest machine support you can:v Capture messages issued by the guest machine itself and route these back to the

ProcOps process for display or automated processing, or bothv Send commands to the guest machine from ProcOps, either as operator requests

or automated actions

Guest Target SystemsThe most likely guest machine that is used as a target system is a LINUX system.When a LINUX machine has a secondary user, the secondary user can use CPSEND commands to:v Issue CP commands to the guest machinev Log on as a user to LINUXv Enter LINUX commands (after logging on)

(It is also possible to set up the LINUX system in such a way that LINUXcommands can be entered on the VM console without logging on to LINUX.)

The secondary user receives:v All "boot up messages"v Responses to CP commands that are run on the guest machinev Responses to logon and LINUX commands

MVS machines are more complex. When an MVS machine is running, the originalVM user first becomes an NIP console and then an MCS console. In these consolemodes MVS takes over all I/O to and from the console, and MVS messages to itcannot be intercepted by any CP facilities. Hence the SCIF SEND command cannotbe used to send commands to MVS, nor can MVS messages to this console beintercepted.

However a "virtual SCLP console" for the guest machine can be used. During theNIP phase of initialization, use of this console can be forced by configuring theguest virtual machine so that it has no usable 3270 consoles. NIP then directs its

Automating Asynchronous Hardware Commands with ISQCCMD and PIPES


messages to the guest machine as line mode commands. This is analogous to thestream of messages sent to the Operating System Messages (OSM) window on anHMC by an MVS system running in a logical partition.

Responses to any NIP messages are entered using the CP VINPUT command.Internally this is done when an ISQSEND command is issued to the operatorconsole (OC) of the target system. To ensure that such VINPUT commands areprocessed correctly, the guest machine must be operating in RUN ON state at thistime.

To ensure that RUN ON state is set, a CP SET RUN ON command is sent to allMVS guest machines at the time when the guest machine is started by the PSM.

Once MCS operation is established, important messages requiring operator actionare directed to the guest machine. Again, these are analogous to the stream ofmessages directed to the OSM window of the HMC. Initially, commands cannot beentered to MVS. To do so, it is necessary to enter "Problem Determination Mode".To enter this mode, a VARY CONSOLE(*),ACTIVATE command must be entered.Once this is done:v All MVS messages that are displayed are routed to the guest machinev Commands may be entered using the CP VINPUT command.

Problem Determination is not generally recommended.

To enter LINUX commands it is normally necessary to log on to LINUX. Thisrequires a user ID and a password. So, to provide for LINUX commands wouldrequire the specification of a user ID and a password to ProcOps, with all theattendant difficulties in the area of security. At present the LINUX system isconsidered IPL COMPLETE when specified messages have appeared. These do notrequire a user logon.

VM machines may also be guest machines. Third level guest machines are notsupported.

VSE machines may also be guest machines.

Customizing Target Systems

LINUXThe LINUX target system should have in its VM Directory entry, a CONSOLEstatement that sets its PSM as its default secondary user. For example, if the virtualmachine LNXAO1 is controlled by a PSM running in virtual machine ISQPSM1,then its CONSOLE statement might be:CONSOLE 009 3215 T ISQPSM1

When a LINUX target system is to be deactivated a FORCE command is used toshut it. The default guest signal timeout interval values (set by the SET SIGNALcommand) and values defined for the guest machine determine the interval usedwhen allowing the LINUX system to shut in an orderly fashion. If this function isrequired for a guest, you must ensure that this is set accordingly.

Such actions may include updating the etc/inittab entry on the LINUX systemitself, and setting up a SHUTTRAP module on the VM host.

VM Second Level Systems Support


MVSThis should have a CONSOLE statement in its VM directory entry that defines itsPSM as its secondary user:CONSOLE 01F 3270 T ISQPSM1

It should also IPL a CMS system as its initial action. Once this CMS system isIPLed it should run a PROFILE EXEC that includes the statements similar to thefollowing:SET RUN ONDETACH 01FIPL 7700

The SET RUN ON is needed so that when a response is to be sent to a NIP consolethe VINPUT command used is effective.

The DETACH is used so that when the MVS system IPLs it finds none of itsdefined 3270 consoles available to it. (You should also ensure that no user issues aVM DIAL to an address that is defined as a NIP or MCS console.)

The IPL command is used to IPL the MVS system.

The MVS system itself should have included in its active CONSOLxx definition aCONSOLE statement for the SYSCONS so that commands can be entered to MVSafter it is IPLed, for example:CONSOLE DEVNUM(SYSCONS)

ROUTCODE(ALL)AUTH(MASTER)MSCOPE(*)CMDSYS(*)UD(Y)

VMThis should have a CONSOLE statement in its VM directory entry that defines itsPSM as its secondary user:CONSOLE 01F 3270 T ISQPSM1

It should also IPL a CMS system as its initial action. Once this CMS system isIPLed it should run a PROFILE EXEC that includes the statements similar to thefollowing:SET RUN ONDETACH 01FIPL 7700

The SET RUN ON is needed so that when a response is to be sent to a console theVINPUT command used is effective.

The DETACH is used so that when the VM system IPLs it finds none of its defined3270 consoles available to it. (You should also ensure that no user issues a VMDIAL to an address that is defined as a Operator Console)

The IPL command is used to IPL the VM system.

The VM system itself should include within its OPERATOR_CONSOLES statementin the SYSTEM CONFIG file (which resides on the "parm disk") a specification forthe emulated system console, for example:OPERATOR _CONSOLES 01F 020 System_Console



This ensures that when VM IPLs and finds no regular consoles available, it thenuses the emulated system console. This in turn directs the messages to thesecondary user as a stream of line-mode messages.

VSEThis should have a CONSOLE statement in its VM directory entry that defines itsPSM as its secondary user:CONSOLE 01F 3270 T ISQPSM1

It should also IPL a CMS system as its initial action. Once this CMS system isIPLed it should run a PROFILE EXEC that includes the statements similar to thefollowing:TERM CONMODE 3215IPL 7700

The TERM CONMODE 3215 command sets the console into line mode.



Chapter 9. Automating USS Resources

This information describes how z/OS UNIX System Services are integrated intoSA z/OS, how to set up z/OS UNIX automation, and provides tips about usingz/OS UNIX automation.

Integration of z/OS UNIX System ServicesThe following functions are supported by SA z/OS for z/OS UNIX applications:v Starting and stopping of applicationsv Monitoring of:

– Processes (represented by the command or path and user ID)– TCP Ports– Files and file systems– Generic User Monitoring (the user supplies a z/OS UNIX monitoring routine

or script)v Using an API to execute z/OS UNIX commands (INGUSS command)

Infrastructure OverviewThe z/OS UNIX resources that should be automated must run in the z/OS UNIXof a z/OS system that is already automated by SA z/OS. From the automationmanager's perspective the NetView agent of this system is responsible for the z/OSUNIX resources.

For command execution through INGUSS or user-defined monitoring, a z/OSUNIX program (provided by SA z/OS) is directly invoked by SA z/OS. Thisprogram (ingccmd) executes UNIX commands and runs when started by SA z/OSwith the jobname INGCUNIX. The ingccmd program is the extension of theNetView-based agent into z/OS UNIX. To monitor the standard z/OS UNIXresources (processes, ports, or files) an internal SA z/OS routine is started.

Process initialization and termination status updates of USS resources are directlyreported from system exits to the SA z/OS environment by theprogram-to-program interface INGUXPPI. A NetView task with the same nameimmediately posts the UP or DOWN status. The automation agent recognizes andthen sets the correct automation status for the resource.

For this functionality, the NetView Subsystem Interface (SSI) is required. For acorrect customization of the SSI, refer to Step 5 in the chapter "Installing SA z/OSon Host Systems" in IBM Tivoli System Automation for z/OS Planning and Installation.

When monitoring of the USS process indicates that it is down, its status is updatedto AUTODOWN. However, because it may take some time before a USS processhas ended (that is, to clean up the resources that is had acquired), monitoring isrepeated after a cleanup delay. If you define your own USS processes, you shouldspecify a suitable cleanup delay using the APPLICATION INFO policy item.Consider using an application class if you need to define several processes.


Setting Up z/OS UNIX Automation

Customization of z/OS UNIX Resourcesz/OS UNIX resources are introduced to SA z/OS by defining them in theSA z/OS customization dialogs.

The customization dialogs support the application type USS. If USS is selected, youcan enter z/OS UNIX-specific data such as a UNIX user ID, command or path,filename, or monitored port. Choose one of these fields to enter the data.

The start and stop definitions can be varied between MVS and z/OS UNIXcommands. For example, to stop an application you can issue a UNIX killcommand first and (if this was not successful) you can perform an MVS cancellater.

Definitions for Automation SetupThe USS path where the program shipped with SA z/OS is located must bedefined in the USS Automation Path field of the System Info policy.

Definitions for z/OS UNIX ResourcesTo define a new application entry (APL, class, or instance), specify the applicationtype USS on the Define New Entry panel. When choosing the application typeUSS, the option USS Control is displayed on the Policy Selection panel.

Select USS Control on the Policy Selection panel to enter the data for the newz/OS UNIX resource. You can specify only the user ID and the z/OS UNIXmonitoring routine for a class on this panel. All other definitions (for example,from/to, dependencies, and so on.) can be entered as usual. For more details aboutthis panel, see IBM Tivoli System Automation for z/OS Defining Automation Policy.

USS applications must be defined with a HASPARENT relationship to JES.

Use the USS Control Specification policy item for an object of type INSTANCE todefine the resource as a:

ProcessEnter the path of the command that is running (as shown by the UNIXcommand ps -e) in the Process Command/Path field.

Filter Enter additional filter criteria to uniquely identify the USS process. Thefield is optional and can only be specified if a 'Command/Path'specification is available.

TCP portEnter the TCP port number that the resource is to listen to on the localhost in the Port Number field.

File Enter the path of a file in the HFS in the File Name field.

IP StackFor TCP port monitoring, you may enter optionally the name of the IPstack if multiple IP stacks are configured.

Often the command/path specification, especially for Java™ processes, is notunique. The Filter field allows you to uniquely identify the USS process if this isnot possible.



There are two methods of monitoring USS applications when using AOFUXMON:v The standard method is to specify the monitoring routine AOFUXMON in the

APPLCATION INFO policy item, which is called by SA z/OS for UNIX SystemServices resources.

v If you choose to use your own script or program in the HFS, this is called byAOFUXMON. You must then specify the script or program in the MonitoringCommand field of the USS CONTROL policy item, and you must also specifyAOFUXMON in the APPLCATION INFO policy item.

If this program does not begin with a forward slash (/) it must reside in the samedirectory as the z/OS UNIX ingccmd routine that is supplied by SA z/OS.Otherwise the name specified is considered to be an absolute path identifier.

The UNIX monitoring routine must have an exit value. It can be one of thefollowing:0 Resource is available4 Resource is starting8 Resource is unavailable12 Error occurred

If the user-specified monitoring routine loops, it receives a SIGKILL after theAOFUSSWAIT time (defined in the NetView stylesheet).

Hint: It is possible to write a message from this UNIX monitoring routine to theMVS system log, in order to trigger an action or perform a status change throughthe NetView Automation Table (AT).

The monitoring routine AOFUXMON must be specified, otherwise the defaultmonitoring routine (usually INGPJMON) is called, which is not sufficient for z/OSUNIX resources.

The Job Type field can be either MVS or NONMVS:

MVS Is only used for resources that represent a process with a unique jobname.For these resources SA z/OS accepts the following messages for statuschanges:v IEF403I Job startedv IEF404I Job endedv IEF450I Job abended

If no start command is specified, the default MVS start method(s <JOBNAME>) is used.

NONMVSSA z/OS ignores the messages listed above for status changes. This isnecessary if the job name is not unique.

For z/OS UNIX resources the Start Delay interval that has been defined beginswhen SA z/OS issues a start command for an application. SA z/OS is informedby z/OS that the resource that is to be monitored has started. This results in theUSS resource being set in the status ACTIVE. After the first start delay interval andsuccessful monitoring, the ACTIVMSG comand is triggered, which sets the agentstatus to UP. The default value for the Start Timeout is 2 minutes.


Chapter 9. Automating USS Resources 109

If you set the Skip ACTIVE status field in the APPLICATION INFO policy item toYES, the resource is immediately set to UP when SA z/OS is informed by z/OSthat the process is running.

For application shutdown, SA z/OS is informed by z/OS as soon as the processhas ended. At this point, SA z/OS immediately sets the resource into theAUTODOWN status.

As a result of this behavior you should carefully consider how you set thefollowing parameters in the APPLICATION INFO policy item, either for theapplication or at the class level:v Start Delayv Start Cyclesv Skip ACTIVE statusv Shutdown Pass Intervalv Cleanup Delay

For more information, see the *USS best practices policy.

Automated Resources

Process Monitoring:

No UNIX process identifiers (PIDs) can be monitored. The monitoring routineneeds the start command and the user ID that the process belongs to. Thisinformation can be obtained with the UNIX command ps.

In the following example all processes that belong to the user USER are displayed:USER:/u/user/ingcmd>ps -e

PID COMMAND33554481 /bin/sh50331698 /usr/sbin/rlogind233554486 /usr/lpp/netview/bin/cnmeunix67108927 /bin/sh83886176 /bin/ps33554821 /usr/sbin/inetd83886472 FTPD67109276 /bin/sh16777629 /usr/sbin/rlogind233554924 HSAPYTCP

This means that automation could not distinguish between the two processesstarted by /usr/sbin/rlogind2. Processes started by identical commands musthave different user IDs.

Alternative 1:If it is necessary to automate processes running multiple instances, a user coulduse softlinks to distinguish between the different processes. For example, theprocess:/u/user/usstest/testme

should be started more than once. In this case, create some softlinks:USER:/u/user/usstest> ln -s testme test1USER:/u/user/usstest> ln -s testme test2

This results in:



USER:/u/user/usstest>ls -altotal 216drwxrwxr-x 2 USER DE#03243 8192 Jan 24 16:24 .drwxr-xr-x 19 USER DE#03243 8192 Jan 24 16:23 ..lrwxrwxrwx 1 USER DE#03243 6 Jan 24 16:24 test1 -> testmelrwxrwxrwx 1 USER DE#03243 6 Jan 24 16:24 test2 -> testme-rwxrwxr-x 1 USER DE#03243 94208 Jan 24 16:23 testme

These three programs (being the same "real" program) can be automated with thethree different start commands test1, test2, and testme. These links may be createdas a prestart command and deleted as a shutfinal command.

Note: Only the command is used, not the parameters that were used to start theprogram. This is because a program may be started by SA z/OS with differentstartup parameters, depending on what the automation manager told theautomation agent to do. In this case, the only constant value is the command, notthe parameters.

Alternative 2:The same program can run in parallel several times by using different startupparameters (like Java programs). In this case it is inefficient to automate theseprocesses as described above. Java programs run in a Java environment and arevisible as Java processes, for example:# ps -e

PID TTY TIME CMD50331734 ? 5h24 .../V6R1/AP/AppServer/java/bin/java83886173 ? 1:44 .../V6R1/AP/AppServer/java/bin/java60341724 ? 2h36 .../V6R1/AP/AppServer/java/bin/java73392173 ? 1:02 .../V6R1/AP/AppServer/java/bin/java

It is impossible in this case to distinguish and evaluate the process that should bemonitored.

The command ps -ef shows the same processes (for example, programs runningin a Java environment), without the fully-qualified Java path but with a parameterchain that is used for startup.#ps -ef

UID PID PPID C STIME TTY TIME CMDEEZDMN 50331734 1 - Jun 27 ? 5h25 java -Djava.util.logging.configureByServer=trueEEZDMN 83886173 1 - Jun 27 ? 1:44 java -Dcom.ibm.eez.adapter.debug=trueEEZDMN 60341724 1 - Jun 27 ? 2h36 java -Djava.util.logging.manager=connectEEZDMN 73392173 1 - Jun 27 ? 1:02 java -Djava.security.auth.login.config=/etc/security.conf

Mapping the output of both commands using the matching PID, a unique processcan be evaluated and monitored. The process that is distinguished is then:/SYSTEM/local/WebSphere/V6R1/AP/AppServer/java/bin/java-Djava.util.logging.configureByServer=true

Where the data that is specified in the UNIX Control Specification panel in theProcess Command/Path field is /SYSTEM/local/WebSphere/V6R1/AP/AppServer/java/bin/java and in the with Filter field is the filter-Djava.util.logging.configureByServer=true.

If the USS program has the sticky bit set, the MVS load is performed using thesymbolic link name. For example, running two instances of syslogd requires theusage of a symbolic link, for example, /tmp/syslogd. A separate /tmp directorymust be used so that the same name (syslogd) can be created.



TCP Port Monitoring:

Exactly one TCP port number can be entered for one resource. SA z/OS monitorsthe local host as returned by the function gethostid().

When this port has a state of 'listening,' this resource is considered to be 'available'in terms of SA z/OS. All other states of the port map to 'unavailable.'

No user ID is required for definitions.

If your system is configured with multiple IP stacks you may specify the name ofthe corresponding IP stack for the defined port in the IP Stack field.

File or File-System Monitoring:

The existence of a file (belonging to a certain user) is verified. Many applicationscreate files at startup and delete these files when terminating normally. If morethan one file should be monitored, this can be modeled as an application group(APG) in the automation manager.

This monitoring can be used to determine if a certain file system is mounted. Thestart command for this resource would be a UNIX 'mount' command, the stopcommand a UNIX 'umount'.

Start and Stop Definitions (INGUSS Command)If the resource is to be controlled by traditional MVS commands, this could bedone in the same way as for all other MVS applications.

Issuing commands in the z/OS UNIX environment is done by specifying theINGUSS command in the start or stop definitions for the resource.

To issue commands in the USS environment use the INGUSS command (for moredetails see IBM Tivoli System Automation for z/OS Programmer's Reference).

Note: INGUSS can only be used if the primary JES is available. Therefore, z/OSUNIX resources using INGUSS need a HASPARENT dependency to JES. Mostz/OS UNIX applications have this dependency. If you want to issue prestartcommands, an additional PREPAVAILABLE dependency is necessary.

z/OS UNIX and MVS commands can be mixed in different shutdown passes.

Command Examples:

Start Command for a Process:To start a process with the command and job name specified in the customizationdialogs, enter INGUSS JOBNAME=&SUBSJOB &SUBSPATH &SUBSFILTER in theCommand Text field on the Startup Command Processing panel of the STARTUPpolicy item of the resource.

Only the command that was used to start an application or a process can bemonitored. If the same program is to be started multiple times, a softlink asprestart command could be used to distinguish the processes.

Use a Softlink to Distinguish Processes That Run the Same Executable File as a PrestartCommand:



To create a softlink for &SUBSPATH (the path parameter of the resource issuingthe command, for example, /u/user1/uss1) and link to the file /u/user1/usstest,enter the following command in the Command Text field on the PRESTARTCommand Processing panel:INGUSS /bin/ln -s /u/user1/usstest &SUBSPATH

When looking at the HFS, this results in:USER1:/u/user1>ls -ltotal 408lrwxrwxrwx 1 USER1 DE#03243 7 Feb 13 12:44 uss1 -> usstest-rwxrwxr-x 1 USER1 DE#03243 163840 Jan 29 14:55 usstest

Stop Commands for a Process:A z/OS UNIX process may be stopped in different ways (using escalation passes).For example, you can first use the z/OS UNIX kill command, if that does notwork use z/OS UNIX kill -9, and finally enter an MVS cancel command.

Enter the definitions for this example as shown in Figure 23 on the CommandProcessing panel for the normal shutdown phase of the resource (via itsSHUTDOWN policy item).

&SUBSPID is replaced at run time by the real PID of the process.

Stop Command for a File:A stop command for a file may be deleting the file. The file name entered in thecustomization dialogs can be found in &SUBSFILE, as shown in Figure 24.

Example: sshdThe Secure Shell Daemon application (SSH daemon or sshd) is the daemonprogram for ssh. This program is an alternative to rlogin and rsh and providesencrypted communications between two untrusted hosts over an insecure network.The sshd is the daemon that listens for connections from clients on port 22.

It is normally started when z/OS UNIX is initialized. It forks a new process foreach incoming connection. The forked processes/connections handle key exchange,encryption, authentication, command execution, and data exchange. Theseconnections show the same jobname and Command/Path and Filter as the SSHdaemon does. At sshd startup time its process ID (pid) is written in the/var/run/sshd.pid file.

Any adaptation and configuration changes to the sshd can be done in the sshdconfiguration file sshd_config. It is located in the /etc/ssh directory.

Cmd Ps AutoFn/* Command Text1 INGUSS /bin/kill &SUBSPID3 INGUSS /bin/kill -9 &SUBSPID4 MVS C &SUBSUSSJOB,A=&SUBSASID

Figure 23. Stop Definitions for a Process

Cmd Ps AutoFn/* Command Text1 INGUSS /bin/rm &SUBSFILE

Figure 24. Delete a File



Keeping the Secure Shell Daemon application (sshd) highly available requires thatthe sshd will not be detached from its parent process. Additionally, the sshd mustbe started in a separate shell environment. This shell is needed to establish aunique process which can be monitored. It can be accomplished by starting thesshd with option -D

For shutdown purposes it is required that the process ID file (sshd.pid) is writtento your file system. This process ID will be read from that file and used to identifythe sshd to terminate.

The ps -ef command supplies further parameters to identify the processreferenced as Filter , for example:ps -ef | grep sshUID PID PPID C STIME TTY TIME CMDAUTO1 83886553 67109368 - 14:53:27 ? 0:00 -sh -c /usr/sbin/sshd -DAUTO1 83886563 50335037 - 14:53:37 ttyp0000 0:00 grep sshAUTO1 83887096 83886553 - 14:53:28 ? 0:00 /usr/sbin/sshd -D

Process 83886553 represents the address space containing the covering parent shellprocess for monitoring purposes. Process 83887096 is the sshd itself.

From this output, set the Filter as -c ’/usr/sbin/sshd -D’ .

Note: Even though the quotation marks are not shown in the output for thecommand ps -ef, they must not be defined in the Filter field of the USS Controlpolicy.

To check for the required information for the Command/Path issue the ps -ecommand and look for the process Id of the parent shell:PID TTY TIME COMMAND...83886553 ? 14:53:27 /bin/sh67109368 ? 14:34:28 BPXBATCH83887096 ? 14:53:28 /usr/sbin/sshd...

or issue ps -e | grep 83886553 to get the process directly:83886553 ? 14:53:27 /bin/sh

This shows the process ID (PID) for the sshd monitoring process. From this output,set the Command/Path as /bin/sh

Next find out the z/OS user ID that the process is running on by issuing thefollowing z/OS command and locating the user ID in the first column where theprocess ID (PID) is listed:D OMVS,PID=83886553

The following output is displayed:BPXO070I 19.01.02 DISPLAY OMVS 035OMVS 000E ACTIVE OMVS=(PA,F9,11,LO,S0,09)USER JOBNAME ASID PID PPID STATE START CT_SECSOMVSKERN SSHD 00D0 83886553 67109368 1WI----- 14.53.27 .0LATCHWAITPID= 0 CMD=-sh -c /usr/sbin/sshd -DTHREAD_ID TCB@ PRI_JOB USERNAME ACC_TIME SC STATE21ADF00000000001 008FF1C0 OMVS .006 WAT W

From this output, set the User ID to OMVSKERN



Verify that sshd listens on port 22 as stated above. Use the netstat -a commandand evaluate the output by looking for the SSHD jobname:# netstat -a...SSHD 00000049 ListenLocal Socket: 0.0.0.0..22Foreign Socket: 0.0.0.0..0...

From this output, set the port to 22.

You can define the UNIX internet daemon (sshd) using the fields of the USSControl policy item for applications (APLs) of type USS in the customizationdialogs with, for example, the data in Table 13.

Table 13. Example Customization Dialog Definitions for sshd

Process File Port

Application Name* SSHD/APL SSH_FILE/APL SSH_PORT/APL

User ID OMVSKERN

ProcessCommand/Path

/bin/sh

Filter -c '/usr/sbin/sshd-D'

File Name /var/run/sshd.pid

PORT 22

IP Stack TCPIP**

* This is the name that was specified for the applications when they were created.** Only ifthe system is configured for multiple IP stacks.

Define a basic group containing all resources with relationships that indicate that:v The group containing all sshd related resources depends on TCPIPv The file is created by the sshd process and can never be started or created

directly by SA z/OSv The sshd process listening on the port can never be started or created directly by

SA z/OS.

Figure 25 on page 116 illustrates the SSHD (modeled as a group) as up andrunning when the process /bin/sh -c ’/usr/sbin/sshd -D' started by userOMVSKERN appears, the file /var/run/sshd.pid exists, and port 22 is in the status'listen' (sshd listens to this port for incoming login requests). You can only choose aport that is defined in /etc/ssh/sshd_config .



Start definition for SSHD_FILE/APLNone.

Start definition for SSHD_PORT/APLNone.

Start definition for SSHD/APLCMD: INGUSS JOBNAME=&SUBSJOB &SUBSPATH &SUBSFILTER (&SUBSxxxvariables are substituted at run time)

Stop definitions for SSHD_FILE/APLThese commands remove the file if it has not yet been removed by thesshd process.

Table 14. Pass description for sshd

Number of Pass Pass Description Command

1 If not done yet by sshd INGUSS /bin/rm &SUBSFILE

4 If Pass 1 did not remove the file INGUSS /bin/rm -f &SUBSFILE

Stop definitions for SSHD/APLWhen stopping the SSHD the correct proccess ID from the sshd.pid filemust be extracted. The kill command is used to terminate the sshd processitself. Once the sshd command is issued the parent shell used formonitoring purposes terminates and the SSHD/APL will go into aAUTODOWN status.

Use the following commands:

CMD:INGUSS /bin/sh -c ’/bin/ps -ef | /bin/grep `/bin/cat/var/run/sshd.pid` | /bin/grep -q sshd && /bin/kill `/bin/cat/var/run/sshd.pid`’

CMD:INGUSS /bin/sh -c ’/bin/ps -ef | /bin/grep `/bin/cat/var/run/sshd.pid` | /bin/grep -q sshd && /bin/kill -9 `/bin/cat/var/run/sshd.pid`’

SSHD

TCPIP

SSH_FILE SSH_PORT

SSH_DAEMON_GROUP

HASPARENT

HASPARENTStartsMeStopsMeMakeUnavailable

WhenDown

ExternallyStartsMe

MakeAvailableWhenAvailableOrStarting

Figure 25. SSH Daemon Listening



CMD:INGRCLUP &SUBSJOB

These commands and the USS pipe are described here:

Table 15. INGUSS Command and USS Pipe Summary

Command Task

INGUSS Is the interface to USS

/bin/sh -c '...' Provides a fully-equipped shell environmentto extract the pid of the sshd

`/bin/cat /var/run/sshd.pid` Subshell extracts the pid of the sshd: theextract pid is represented by 'extracted_pid'in the left column

/bin/ps -ef Asks for all USS processes

| /bin/grep extracted_pid Shows only the sshd process

| /bin/grep -q sshd Suppresses output and returns theappropriate return code: return code 0: sshdprocess exists return code 1: sshd processdoes NOT exist

&& Continues only if return code is 0 / if sshdexists

/bin/kill extracted_pid or /bin/kill -9extracted_pid

Issues SIGTERM signal against the 'extractedpid' of the sshd

In case, the kill command does not terminate the sshd process useINGRCLUP routine to invoke a z/OS CANCEL command against theaddress spaces of the sshd and the monitored parent shell.

SA z/OS provides the *USS best practices policy that provides definitions forseveral automated USS daemons, such as sshd. Common definitions for USSresources can be found in the APL classes starting with C_USS_xxx .

Use also the Unix man pages to get more information about the used USScommands and their parameters.

Hints and TipsUSS tasks behave differently when started as STCs rather than directly in the USSenvironment.

When a USS task is started as an STC, the starting user ID may differ so that, inmost cases, the AOFUXMON monitor routine is not able to internally triggerACTIVMSG UP=YES.

In this case it is much simpler for SA z/OS to start these applications withINGUSS. An AT entry is then not required for the UP message. SA z/OS is able tointernally simulate this so that you do not have to worry about UP messages.

Job names (that is, the last character of the job name) are not predictable for USSresources. However, AOFUXMON is able to handle this by monitoring the pathwithin USS and changing the defined job name in SA z/OS accordingly.

For the syslog daemon you would define the job name as SYSLOGD. When theapplication is started and changes the job name to, say, SYSLOGD7, AOFUXMONadjusts the SA z/OS data model to reflect this. However, this cannot be handled in



|||

|||

|||

|||

the AT with a generic entry for SYSLOGD*. This is because the change in the jobname is caused by the USS process that creates a new address space with a newname, so that the old address space with the old name terminates. This means thatyou get an ended message for the old address space and an UP message for thenew address space. The sequence of these messages is also unpredictable.

Trapping UNIX syslogd MessagesTo trap UNIX syslogd messages, an entry must be added to the syslogdconfiguration file /etc/syslog.conf in order to forward the messages to the MVSsystem log. Thus, messages can be processed by the NetView automation table(AT).

To forward all messages to the MVS log add the following entry:*.* /dev/console

To send special messages to the MVS log only, follow the syslog message namingguidelines (for example, for warning messages use *.warn). You can use/dev/console as an ordinary file to write to.

To test this, issue the following UNIX command from a USS console:echo ’This is a test message’ >>/dev/console

The UNIX messages have the MVS message ID BPXF024I and are multilinemessages.

Figure 26 shows an example of the output of the UNIX command in the systemlog.

DebuggingDebugging can be activated for z/OS UNIX monitoring and command executionon the AOCTRACE panel. The automation procedure for monitoring isAOFUXMON and for command execution AOFRSUSS.

Turning on debugging for AOFRSUSS implicitly turns on debugging for ingccmd(the SA z/OS command server).

The debugging messages are written to the netlog and to the z/OS UNIX systemlog (syslogd).

M 13:45:21.34 STC03602 00000090 BPXF024I (USER) Feb 13 13:45:21 SYS1 syslogtest 67109100 : This isS 498D 498 00000090 a test message

Figure 26. Example of a UNIX Message

Hints and Tips


|||||

Chapter 10. Command Receiver

This information describes how to set up and use command receivers for generaluse. The command receiver is able to pass NetView, SA z/OS or MVS commandsto SA z/OS for execution. The command receiver is able to dispatch incomingcommands to a dedicated group of tasks, called the command work tasks. Thecommand receiver uses a round robin algorithm that ensures, on average, each ofthe associated automated functions has the same number of commands to process.In short, the technique enables the execution of multiple received commands inparallel.

The command receiver is used by:v the Batch Command Interfacev AT Overwrite Syntax Checking of the Customization Dialogv Relational Data Services from TSOv other SA z/OS provided REXX functions running in TSO.

Setting Up the Command ReceiverSetting up the command receiver consists of the following:1. Setting up the TSO/Batch environment, see the installation steps in “Use RDS

for TSO” on page 128.2. Defining the command receiver as a subsystem automated by SA z/OS3. Defining the command work tasks used by the command receiver.

It is recommended to use the SA resources defined in the add-on policy*IBMCOMP. Select the following entries:

Table 16. Policy Entry Names and Types for Command Receivers

Policy Entry Name Policy Entry Type

CMDRCVR APL

CMD_RECEIVER APG

CMD_RECEIVER_AUTOOPS AOP

These SA z/OS resources ensure that the command receiver task is started atSA z/OS initialization time and ready to receive commands. The operator is ableto monitor the status of the command receiver by means of the INGLIST commandvia SDF.

Setting Up TSO/Batch EnvironmentAdd REXX function package INGTXFPG in the appropriate TSO module. TSO/Eprovides the following samples in SYS1.SAMPLIB that you can use to code yourown load modules:

Table 17. Sample Names and Load Modules for a TSO/Batch Environment

Sample Name Load Module name

IRXREXX1 IRXPARMS for MVS

IRXREXX2 IRXTSPRM for TSO/E


|

|

|

|

||

Table 17. Sample Names and Load Modules for a TSO/Batch Environment (continued)

Sample Name Load Module name

IRXREXX3 IRXISPRM for ISPF

There are various considerations for providing your own parameter modules. Fordetails, see the section "Function Package Table" in chapter "Language ProcessorEnvironments" of TSO REXX Reference. The different considerations depend onwhether you want to change a parameter value only for an environment that isinitialized for:v ISPFv both TSO/E and ISPF sessionsv a non-TSO/E address space

Select the appropriate sample parameters modules, for example, IRXREXX2 forTSO/E also for batch PGM=IKJEFT01, and add function package INGTXFPG as asystem function package.

The following must be considered:1. You must link-edit the REXX default parameters module with the

corresponding names. For example, the load for the sample IRXREXX2 musthave the name IRXTSPRM.

2. Place the resultant REXX default parameter module in the LPALST.3. Make sure that also the function package INGTXFPG resides in the Linklist.

Defining Command Receiver as a Subsystem Automated by SA z/OSImport the CMDRCVR and CMD_RECEIVER objects from the sample policy*IBMCOMP. This creates an application group containing a non-MVS subsystemcalled CMDRCVR presenting the command receiver. The command receiver taskwill run in a VOST. The name of the VOST is the name of the associated resource,in this case CMDRCVR.

When the CMDRCVR starts, it starts the PPI receiver by means of command:INGRCRCV START OPF=AOFCMDOPER

The default PPI receiver identifier is INGRCRCV, but can be changed to any othername. If you want to use a different PPI receiver ID you must specify theparameters PPI and CSAKEY. The CSAKEY parameter defines a key that allcommand receiver clients are using for communicating with the receiver.EMULATOR is the only key supported. The INGRCRCV command would looklike:INGRCRCV START PPI=xxxxxxxx CSAKEY=EMULATOR OPF=AOFCMDOPER

where xxxxxxxx is the PPI name of choice.

Be aware that the command receiver is not able to route the incoming command toone of the command work tasks when the OPF=AOFCMDOPER parameter ismissing. In this case, all incoming commands are processed sequentially by theVOST task hosting the command receiver.

When subsystem CMDRCVR terminates, it stops the command receiver by meansof INGRCRCV STOP command.

Command Receivers


|

Notice that the sample definition has escalation commands defined for the stopprocess. It uses the INGVSTOP DETACH and INGVSTOP TASK commands for thestop passes. These commands stop the VOST hosting the command receiverdirectly rather than terminating the command receiver gracefully.

Defining Command Work Tasks Used by the Command ReceiverImport object CMD_RECEIVER_AUTOOPS from the sample policy *IBMCOMP.This object import defines a set of automated functions named AOFCMDnn wherenn starts from 1 running up. By default 5 command work tasks are defined. Youmay add more command work tasks if needed.

Notes:

1. The command receiver checks if the automated functions (command worktasks) are defined and active. If this is not the case, the command receiverterminates with error message ING347I.

2. Note that the names of NetView Automation Operators must match those thatyou define in member ING.SINGPRM(AOFOPFSO). You may customize thissample to your needs. By default, the following map is used:

Table 18. Functions and Operators for SA and NetView respectively

SA Automated Function NetView Automation Operators

AOFCMD01 | AOFCMDnn AUTCMD01 | AUTCMDnn

Starting and Stopping the Command ReceiverThe command receiver is controlled by SA z/OS. It is defined as a non-MVSsubsystem. Unless changed, the name of the command receiver subsystem isCMDRCVR.

To start the command receiver, issue the INGREQ REQ=START command againstthe appropriate subsystem, for example:INGREQ CMDRCVR/APL/SYS1/ REQ=START

You should not have to start the command receiver in normal circumstances,because it should start automatically when the SA z/OS agent registers with theautomation manager.

To stop the command receiver, issue the INGREQ REQ=STOP command againstthe appropriate subsystem, for example:INGREQ CMDRCVR/APL/SYS1 REQ=STOP

To see the list of VOSTs used by the command receiver(s) use the INGRCRCVQUERY command.

Submitting NetView Commands from a Batch JobThis section describes how to execute NetView commands from a Batch job. This isparticularly useful for Tivoli Workload Scheduler, but can be used stand alone.

Sample Batch Job JCLA sample batch job can be found in the System Automation for z/OS Installationlibrary SINGSAMP. Member EVJSJ001 contains the sample JCL.

Command Receivers

Chapter 10. Command Receiver 121

The batch job must be run on the same system as the SA z/OS Agent that containsthe command receiver specified by the batch job. In most cases there will be acommand receiver running on every SA z/OS Agent. The use of the generalpurpose command receiver is recommended since it can handle the execution ofmultiple commands concurrently. The use of the general purpose commandreceiver is triggered by the SERVER=* parameter. However, customization of thecommand receivers can alter the names of the command receivers and also thenumber and configuration of the command receivers. You should check with yoursystem programmers to determine the correct system and command receiver to usefor these batch jobs.

Command Statement SyntaxThe commands supplied to the batch job in the //SYSIN ddname have thefollowing syntax:

►► line-mode-command>DDNAME "-"

*comment

►◄

1. All blank lines are ignored.2. All lines starting with an asterisk (*) are comment lines and are printed in the

output but otherwise ignored.3. Comments on the end of commands are not allowed.4. Comments are not allowed between continuation lines.5. A command can be continued by appending a dash (-) to the line.6. Command output normally goes to //SYSTSPRT.7. Command output may be redirected to other DDNAMEs. The default for this is

the right angle bracket (>) symbol.8. PIPE > stage is prohibited. Use PIPE QSAM instead.9. Fullscreen commands are not allowed.

Valid Command TypesAny command, clist or REXX program that issues correlated line messages may beused.

This means almost all NetView commands, all SA z/OS commands that supportOUTMODE=LINE and any clist or REXX program that either issues SAY messagesor PIPES the messages to CONSOLE.

The return code from the command can be used to stop the remaining commandsfrom being executed. See the MAXRC parameter of the AOFRYCMD proceduredefinition.

Command ContinuationCommands are continued across lines by appending a dash to the end of thecommand, for example:PIPE NETVIEW LIST STATUS=OPS | -CONSOLE ONLY

Command Output RedirectionNormally command output is printed on the //SYSTSPRT DDNAME. However,the output of commands may be redirected to other DDNAMEs. This is achievedvia a redirection symbol. The default is the > symbol, for example:

Sample Batch Job JCL


PIPE NETVIEW LIST STATUS=OPS | CONSOLE ONLY >MYOUTPUT

This allows subsequent steps in the batch job or other batch jobs to use the outputof the command for their own purpose.

You can change the redirection symbol with the REDIRECT parameter ofAOFRYCMD if, for example, you use > as a command prefix. Note that theredirection symbol must not be the same as any of the characters that occur in thecommand. For more details, see “AOFRYCMD Description.”

The DCB characteristics of the output DDNAME should be as follows:LRECL=132,RECFM=FB

Command Information Retrieval

A REXX program scheduled with the command AOFRYCMD (or by deprecatedEVJRYCMD) is able to retrieve the originating jobname and user id through taskglobals:

jobname = TGLOBAL('AOFRYCMD_JOBNAME')userid = TGLOBAL('AOFRYCMD_USERID')

Note: When using the general purpose receiver by coding SERVER=* thenTGLOBAL AOFRYCMD_USERID contains the TSO userid, populated by userid()function in AOFRYCMD.

When not using the general purpose command receiver then, the TGLOBALcontains the SAF ID of the PPI sender, populated by the PIPE EDIT SESSID stagein command slave EVJRVCMD.

Executing a Command on a Different NetViewAlmost all SA z/OS commands can specify the TARGET= parameter to force thecommand to execute on the target system.

If a command does not have this facility, for example the NetView LIST command,you can use PIPE labels to send the command to the appropriate NetView, forexample:PIPE CC dom01: LIST STATUS | CONSOLE ONLY

Or even:PIPE CC dom01/auto1: LIST STATUS=OPS | CONSOLE ONLY

JCL for the Batch Command InterfaceYou will find the sample JCL EVJSJ001 to execute commands onSA z/OS/NetView agents in the SA z/OS product sample library (SINGSAMP).

AOFRYCMD Description

Purpose: AOFRYCMD is a REXX procedure that issues commands to a SA z/OSagent and receives the results of those commands.

Note: AOFRYCMD is identical to EVJRYCMD. For an easy migration EVJRYCMDstill exists in the library SINGNREX while AOFRYCMD resides in SINGTREX. Theuse of AOFRYCMD is recommended.

Command Statement Syntax


|

|||

|

|

|||

|||

►► AOFRYCMDwsid NOWKSTS

SERVER = EVJCMDRV

SERVER = name*

►

►TIMEOUT = 60

TIMEOUT = secondsTIMEOUT = NONE

HIGHRC = 0

HIGHRC = return_code►

►MAXRC = 999

MAXRC = return_code

SYSIN = SYSIN

SYSIN = DDname

REDIRECT = >

REDIRECT = chars►

►ASIS = NO

ASIS = YES►◄

Parameters:

wsid This parameter is optional.

This parameter specifies the name of the TWS workstation that submittedthis batch job. This information is used by the command to disable theworkstation in the event that communications between the batch job andthe SA z/OS Agent cannot be established.

If this parameter is specified, any NetView PPI communications problemwill cause the command to issue a TWS WSSTAT command to place theworkstation offline.

NOWKSTSThis parameter is optional. It is also deprecated. It is preferable to omit theworkstation ID.

This parameter modifies the behavior of the command. In the event of afailure in communications to the SA z/OS Agent, this parameter preventsthe command from disabling the TWS workstation that is defined with thewsid parameter.

SERVERThis parameter is optional.

The default for this parameter is EVJCMDRV. This parameter specifies thename of the PPI receiver in the SA z/OS Agent NetView that commandswill be sent to. Specifying SERVER=* causes the command receiver to passthe command to one of the associated work tasks to enable parallelprocessing of commands. The command receiver EVJCMDRV is deprecatedbut still exists for compatability reasons.

For details on defining EVJCMDRV, refer also to the *IBMCOMP Add-onpolicy.

Note: When using the general purpose command receiver by codingSERVER=*, the SINGTREX library must be added to the SYSPROCconcatenation chain.

TIMEOUTThis parameter is optional.

The default for this parameter is 60 seconds.

JCL for the Batch Command Interface


|||||||

This parameter specifies the time in seconds that the batch job will wait fora command to execute in the SA z/OS Agent NetView. This timeout isapplied separately to each command. If the timeout is set to NONE, notimeout will be applied to the batch job.

Note: It is recommended that the INGREQ timeout (as defined with theFDBK parameter) should be less than the TIMEOUT= parameter for thejob.

This is because the INGREQ command's FDBK parameter can be used tospecify a WAIT period that will result in the command waiting until thedesired status change is complete. For example, if the TIMEOUT parameteris defaulted to 60 seconds, the INGREQ FDBK parameter should be codedas, say, FDBK=(WAIT, :55)

HIGHRCThis parameter is optional.

The default for this parameter is 0 (zero).

This parameter specifies the highest acceptable Return Code for the job.Any return codes from commands that are less than or equal to this valuewill reset the JCL Step return code to zero. Any command return code thatis greater than this value will be passed as the JCL Step return code.

Note: The JCL Step return code will be the highest return code of all thecommand return codes.

MAXRCThis parameter is optional.

The default for this parameter is 999.

This parameter specifies the maximum acceptable return codes fromcommands issued by the batch job. If a command return code is higherthan the value specified, the batch job is aborted and any remainingcommands will not be executed.

The return code that is reported to the JCL is determined by the HIGHRCparameter.

SYSINThis parameter is optional.

The default for this parameter is SYSIN.

This parameter sets the DDNAME of the input file that contains thecommand to be executed.

REDIRECTThis parameter is optional.

The default for this parameter is >.

This parameter defines the redirection character. Enclose it in quotes ordouble-quotes if the string contains special characters, such as the equalsign. It must not be the same as any of the characters that occur in thecommand.

ASIS This parameter is optional.

The default for this parameter is NO.



This parameter enables the submission of a required command in mixedcase, to be executed as presented, when the parameter value is YES.

Usage:When the SA z/OS Agent is started, it will automatically issue a WSSTATcommand to mark the workstation online. The specifications of which workstationsto mark online at agent restart are contained in the WORKSTATION message/userdata policy for the tracker or controller. Multiple workstations may be defined.Workstations that are assigned to trackers should have their WORKSTATIONpolicy defined to the same trackers that they are assigned to.

Each command is submitted in turn and the results of the command are retrieved.These results are then written to either SYSTSPRT or to the output redirectionDDNAME.

Whether or not the command completed successfully is indicated by messageING330I for satisfactory completion or message ING332I when the commandfailed. Message ING332I shows the return code of the command completion. Thesemessages are written to SYSPRINT (//SYSTSPRT DDName).

Note: The DW0369I message that might have been requested when using theNetView PIPE command with the MOE option or automatically injected by SAwhen the command completed successfully is no longer passed along.



Chapter 11. Enabling Relational Data Services (RDS)

SA z/OS provides Relational Data Services (RDS) which are available through theINGRDS command. It provides basic access methods for SA z/OS built-inrelational data tables. The syntax of the INGRDS command is similar to the API ofSQL but without the full SQL language parser. However, RDS is not a full SQL anddoes not support all concepts of SQL. SA z/OS saves the relational data tables inVSAM files.

For fast access, the relational data tables are held in the GETMAIN storage of theNetView address space. With PTF UA54030, the relational data service tables areheld in 64-bit storage. Only the control data is still stored in 31-bit storage of theNetView address space. This increases the capacity of user data stored into theRDS tables. Persistence is achieved by an algorithm that periodically saves thetables into a VSAM KSDS file with DD INGEMUGL The tables will be restoredduring SA initialization when NetView is started.

Each SA z/OS agent maintains its own VSAM data file and in memory tables thatonly hold data associated with it. When the SA z/OS agent is down, the table datacannot be accessed.

Note: There is one VSAM KSDS file per SA/NetView agent which holds thepersistent RDS tables. Accessing the RDS tables is only possible via INGRDS if theSA/NetView is running.

Enable/Disable Persistent Relational Data TablesRelational Data Services (RDS) are available at any time after System Automationinitialization. However, new or modified RDS tables are persistent only if thearchiving application RDSARCH runs periodically in the background or if yousave manually a specific RDS table via command INGRDS ARCHIVE.

Prerequisite for archiving and restoring RDS tables is the existence of the VSAMfile INGEMUGL. RDS tables are restored during System Automation / NetViewstartup. Refer to the step "Install Relational Data Services" in the chapter "InstallingSA z/OS" of IBM Tivoli System Automation for z/OS Planning and Installation Guide.

Once RDS is initialized you may disable making RDS tables persistent by stoppingthe resource RDSARCH. This stops archiving the RDS tables. However RDS stillworks with the in-memory RDS tables.

Import System Automation ResourcesIt is recommended you use the SA z/OS resources defined in the add-on PDB*IBMCOMP.

Select the following entries:

Table 19. Receiver Names and Related Policies


RDSARCH APL

RDS_ARCHIVER APG


|||

||||

|

||||

||||

|||

Table 19. Receiver Names and Related Policies (continued)


RDS_AUTOOPS AOP

These SA resources ensure that the RDS archive task makes the RDS tablepersistent. The advantage is to start or stop archiving easily. Error situationsreflected by error messages such as VSAM IO error could also be trapped in themessage table and associated with the SA resource. The resource status mightbecome broken indicating a severe error.

After importing the RDS_ARCHIVER and RDSARCH, the SA z/OS customizationdialog defines an APG resource group that includes an APL resource RDSARCH. Itprovides the function to start or stop RDS archiving. Using INGTIMER as aPRESTART and REFRESHSTART command, the RDS archiving command will bescheduled on the automated function AOFRDSAR periodically every nn seconds.See “Regular Snapshot.”

After importing RDS_AUTOOPS the SA customization dialog defines thecorresponding automated functions AOFRDSAR and AOFRDSEV. Note that thenames of the NetView Automation Operators must match those that you define inmember ING.SINGPRM(AOFOPFSO). You may customize this sample to yourneeds. By default the following mapping is used:

Table 20. RDS/SA Functions and NetView Operators

SA Automated Function/Operator NetView Automated Operator

AOFRDSAR AUTRDSAR

AOFRDSEV AUTRDSEV

Regular SnapshotBy default, the RDS archiving resource RDSARCH issues the command"INGVALUE ARCHIVE" every 30 seconds. It runs on the automated functionAOFRDSAR. Archiving and restoring must run always on the same task. Makesure that the RDS archiving works well and periodically performs the backup ofRDS tables.

RDS InitializationDuring SA z/OS initialization the persistent RDS tables are restored into memory,if the VSAM file with DD INGEMUGL exists.

If this status of the RDS initialization is not OK then the following message mayappear:ING388I Function or command INGRCVAC failed, RC=36 REASON=ARCHIVE rejected INIT STATUS=xxxx

If xxxx is a NULL string then the initialization of RDS was not performed.

Use RDS for TSOThis section provides some general guidelines for System Automation (SA)Relational Data Services on TSO.

Import System Automation Resources


||||||

|

||

You may execute the functions provided by INGRDS also under TSO. For thatpurpose use the TSO client program INGRCRDX, for example, INGRCRDX QUERYmytable.

This client program executes the command INGRDS under cover on the local SANetView Agent and receives the response. Before using INGRCRDX you mustperform some installation steps:v Setup the TSO environment for use by the command receiver

– Install the SA provided TSO REXX function package INGTXFPG. For moreinformation, refer to the step "Configure Function Packages for TSO" in"Traditional SA z/OS Configuration" in IBM Tivoli System Automation for z/OSPlanning and Installation.

– Make sure the TSO programs are able to execute the REXX librarySINGTREX.

– Install the TSO authorized command INGPAUTH. For more information, referto the step "Configure Function Packages for TSO" in "Traditional SA z/OSConfiguration" in IBM Tivoli System Automation for z/OS Planning andInstallation.

v Install the command receiver in SA / NetView– The command receiver must be installed and started on the local SA NetView

agent. For more information, see Chapter 10, “Command Receiver,” on page119.

v Optionally setup security profilesv For diagnostic purposes, you may verify the steps above by calling TSO

command:INGRCRDX CKINST

If you set up RDS for TSO successfully the command output should look similarto:SA TSO function package timestamp: 2014167 143531SA TSO authorized command OKSA TSO RPC to NetView OKSA TSO RPC received VERSION=V3R5M0 GRPID=AA

Setup Security Profiles

Optionally you may install SAF security. The corresponding SAF user id is alwaysthe originating TSO user id.

There is a SAF front-end checking which is done, by INGRCRDX on TSO:READ ACCESS required for SYSAUTO and profileTSO.sysplex.system.CMDRCVR.SEND

There is a SAF back-end checking which is done by the command receiver onNetView:

READ access required for CLASS NETCMDS and profilenetid.domain.INGRCRDS

Use RDS for TSO

Chapter 11. Enabling Relational Data Services (RDS) 129

|||

|||

|

||||

||

||||

|

|||

|

||

|

||

||||

|

||

|

|

|

||

|

|

RDS Table EditorSystem Automation provides support for the ISPF editor on TSO in order to edit aRDS table and to save changes made at the RDS table. While editing a RDS table itis protected against modification by other users via a LOCK token. The lock tokenis JOB$xxx where xxx is the TSO user id.

Note: If you have defined the data set with variable records you may get thewarning message below. It is acceptable that all trailing blanks are removed.Saving the table again will work anyway. “Truncation warning. The data you areediting is variable length data with at least one record that ends with ablank. Saving the data will result in removal of any trailing blanks fromall records. ...”

For an ISPF EDIT example see screenshot below:

000001 @SA_RDS_TABLE: TABLE1 (30,20,15)000002 NAME FIRST_NAME CITY000003 ---------------------------------- --------------------------------- --------------000004 Bond James London000005 Smith John New York000006 Schulz Emma Berlin000007 Metzger NULL München000008 Wagner Frankfurt000009 Schiller Eva Stuttgart000010 Schmidt Hugo Hamburg000011 Fischer Egon Hamburg****** ********************************* Bottom of Data ***********************************

sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssNTruncation warning. The data you are editing is variable length data with at eleast one record that ends with a blank. Saving the data will result in eremoval of any trailing blanks from all records. You can issue the PRESERVE eON command if you don’t want the blanks removed. esssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssm

Note: Trailing blanks can be removed to save space.

Customize the TSO EnvironmentAdditional installation steps need to be performed. See “Use RDS for TSO” onpage 128.

Define a RDS Working Data Set for Viewing/Editing under TSOViewing and editing a table is done by use of a temporary member of a dedicatedPDS. NetView and TSO must have read and write access to this PDS. The PDS isreserved for this function. It must have maximum record length big enough tosupport table rows with longest length that fit to your needs.

A unique temporary member will be created on behalf of the TSO user that callsINGRDS EDIT or VIEW. For that purpose the TSO user id is used. The memberwill always be overwritten with subsequent calls to INGRDS EDIT or VIEW.

The RDS working data set must be made know to all TSO address spaces and toNetView via following AAO:AOF_AAO_RDS_TSO_DSN=HLQ.RDS.WORK

The data set name HLQ.RDS.WORK is customizable. You may create a PDS with aname of its own.

RDS Table Editor


|

||

Sample of a PDS characteristics with maximum records length of 1000 if the totalsum of all columns definition of a table is smaller 1000 bytes:Organization . . . : PORecord format . . . : VBRecord length . . . : 1000Block size . . . . : 32000Data set name type : PDS

Viewing a RDS Table within TSOYou use the following command on TSO:INGRCRDX VIEW mytabINGRCRDX BROWSE mytab

It retrieves the table mytab and displays it using the ISPF editor.

Editing a RDS Table via TSOYou use the following command on TSO:INGRCRDX EDIT mytab

It retrieves the table mytab and displays it using the ISPF editor. You may changethe table. Please observe the rules of column specifications as described below.Changes to the table will be saved in a temporary file and imported to the RDStable mytab. A table lock is obtained until the table is saved or quit.

The following rules apply:1. Respect the column length and keep one blank between each column.2. For an existing table you should not delete columns or insert new columns

because you cannot overwrite column definitions.3. For a table that does not exist yet: After you modified the table name in line 1

and saved the table you can delete, insert and rename columns. Make sure thatthe numbers in parenthesis match the column width. The new table name mustnot exist. If the new table name exists already the old column definitions areused.

Define a RDS Working Data Set for Viewing/Editing under TSO

Chapter 11. Enabling Relational Data Services (RDS) 131

Define a RDS Working Data Set for Viewing/Editing under TSO


Chapter 12. Enabling Sysplex Automation

This information covers SA z/OS specific capabilities for Parallel Sysplex®

automation, how to use the SA z/OS customization dialogs to enable them, andhow to customize your system.

Note: If you use a host code page other than 037, the hexadecimal representationof the at sign (@) can be different. Use the letter represented by the hex code X'7C'for the at sign.

Sysplex FunctionsThe following functions are described:v “Managing Couple Data Sets”v “Managing the System Logger” on page 134v “Managing Coupling Facilities” on page 135v “Recovery Actions” on page 137v “Hardware Validation” on page 144

Managing Couple Data SetsCouple data sets (CDSs) contain control information about the sysplex and itsresources, and are of crucial importance for the functioning of a Parallel Sysplex.

Particularly important are the SYSPLEX couple data set, which containsinformation about the systems and the communication structure (XCF groups) ofthe sysplex, and the CFRM couple data set, which specifies its coupling facilities(CFs) and structures (see “Managing Coupling Facilities” on page 135). Every MVSsystem in a Parallel Sysplex must have access to these CDSs, and to those of allother implemented sysplex functions, such as SFM and Application ResponseMeasurement (ARM).

If a member system cannot access a CDS, the corresponding sysplex function isimpacted, and in some cases the sysplex goes down. It is therefore recommendedthat you define two CDSs to XCF for every CDS type required for theimplementation of the sysplex. One of these, the primary CDS, is the one that isactually used. The other, which is called the alternate CDS, serves as a backup copy.The two CDSs contain the same data. Whenever the primary CDS changes, XCFupdates the alternate CDS accordingly. If an alternate CDS is available for a certaintype, XCF automatically switches to this alternate CDS whenever a member can nolonger access the primary CDS.

All CDSs except the sysplex couple data set contain one or more user-definedconfigurations, called policies. For each CDS type, only one policy can be active.However, it is possible to switch the active policy at run time. Refer to IBM TivoliSystem Automation for z/OS Operator's Commands for further information about theINGPLEX command.

SA z/OS offers two functions for easier CDS management:v Automated creation and recovery of alternate couple data sets for continuous

availability


v INGPLEX CDS, which simplifies management of couple data sets

Ensuring Continuous Availability of Couple Data SetsWhen an alternate CDS exists for a given CDS type and the current primary CDSfails, XCF makes this alternate the primary CDS. After this switch, however, analternate CDS no longer exists, and if the current primary CDS also fails, theproblems that were to be avoided by the creation of an alternate occur again. Toavoid this single-point-of-failure situation, SA z/OS provides a recoverymechanism that tries to ensure that an alternate CDS is always available for everyCDS type used.

SA z/OS creates a new alternate CDS in the following two situations:v During initialization, SA z/OS checks that an alternate CDS is specified for

every primary CDS. If there is a primary CDS for which no alternate CDS exists,SA z/OS automatically creates it.

v At run time, SA z/OS ensures that a new alternate is created whenever thecurrent alternate has been removed or switched to the primary one.

CustomizationRecovery of alternate CDSs is initiated either by the CDS function of INGPLEX orin the background (for example, at initialization time). Background recovery can beswitched on and off by using the SA z/OS customization dialogs. Automaticre-creation with INGPLEX CDS is always enabled.

You must specify the spare volumes that SA z/OS may use for creating missingalternate CDSs (using the policy item SYSPLEX from the Policy Selection panel forsysplex groups). This is also required for automatic creation with INGPLEX CDS.Every CDS type has its own pool of spare volumes. Note that if you do not definespare volumes for a CDS type, no recovery is performed for this type. For detailson the use of the customization dialogs, see “Enabling Continuous Availability ofCouple Data Sets” on page 147.

You can control access to those functions of INGPLEX CDS that modify the sysplexconfiguration. Refer to the "Security and Authorization" chapter of IBM TivoliSystem Automation for z/OS Planning and Installation for details.

Managing the System Logger

Terms and ConceptsThe system logger provides a sysplex-wide logging facility. Applications that use thesystem logger write their log data into log streams. Within a Parallel Sysplex, theselog streams are usually associated with a coupling facility structure. For furtherinformation about coupling facility structures, refer to “Managing CouplingFacilities” on page 135. By using a coupling facility log stream, members of amultisystem application can merge their logs even when residing on differentsystems.

When an application writes data to a log stream this data is stored at firsttemporarily in the associated structure (coupling facility log stream) or a localbuffer (DASD-only log stream). From there, it is off-loaded into a log stream dataset which is automatically allocated by the system logger. When this log streamdata set is full, the system logger allocates a second one, and so on.

The control information for the system logger, which includes a directory for thelog stream data sets of every log stream, is contained in the LOGR couple data set.

Managing Couple Data Sets


The total number of log stream data sets that can be allocated by the system loggeris determined when the LOGR couple data set is formatted.

Two problems that can arise in connection with the log stream data sets are ashortage of directory space in the LOGR CDS and incorrect share options for thelog stream data sets. SA z/OS provides the following recovery actions for theseproblems:v The primary and alternate LOGR CDSs are automatically re-sized if there is a

directory shortagev The operator is notified if the share options for log stream data sets are not

defined correctly

Resizing the LOGR Couple Data Sets in Case of DirectoryShortageThe LOGR CDS contains information about the log stream data sets used by thesystem logger. This information is stored in directory extents. Every directory extentrecord can hold information about up to 168 log stream data sets. The number ofdirectory extents available in a LOGR CDS is specified when the CDS is formatted(DSEXTENT parameter). When all available directory extents are used up the systemlogger can no longer allocate new log stream data sets. This can cause considerableproblems for applications that use the system logger.

With SA z/OS, you can avoid this situation. If you switch on logger recovery,SA z/OS automatically reformats your primary and alternate LOGR CDS with anincreased DSEXTENT parameter whenever the system reports a directory shortage.

CustomizationAutomation of system logger recovery is enabled through the SA z/OScustomization dialogs. For more details, see “System Log Failure Recovery” onpage 191.

Managing Coupling FacilitiesA coupling facility (CF) is a logical partition that provides storage for data exchangebetween components of an application that is distributed across different systemsin a Parallel Sysplex. A Parallel Sysplex can contain more than one CF.

The storage of a coupling facility is divided into areas that are called structures.You can imagine a structure as a special kind of data set. It is these structures,which are identified by their name, that are accessed for reading and writing bythe application components.

The association between CFs and structures is dynamic. A structure that is used byan application need not be allocated at all (for example, when the application isnot running), and can be allocated on different CFs at different points in time. Forevery structure, there exists a preference list that defines the CFs on which it may beallocated. The order of the CFs in that list determines which CF is selected whenmore than one member of the list satisfies all allocation requirements (for example,provides enough space).

The preference list, the space requirements, and other properties of the structuresare defined in the active CFRM policy. This policy is contained in the CFRMcouple data set. Refer to “Managing Couple Data Sets” on page 133 for furtherinformation.

Managing the System Logger

Chapter 12. Enabling Sysplex Automation 135

XES allocates a structure that does not yet reside on any CF when an applicationcomponent needs to be connected to it. Note that the application component onlyspecifies the name of the structure that it wants to access. It is XES that decides onwhich CF the structure is allocated. This decision is influenced by the structuredefinition in the active CFRM policy. After the structure has been allocated, therequesting application component can access it, and further components of thisapplication can require to connect to it. An application component that has accessto an allocated structure is referred to as an active connector to this structure.

In the simplest case, XES deallocates a structure when all connected applicationcomponents have disconnected from the structure. However, an applicationcomponent can require that the structure or its own connection to the structure bepersistent. When the structure is persistent it remains allocated even when theapplication component is no longer connected to it. When a connection is persistentthe structure remains allocated after a failure of that connection. The applicationcomponent in question remains a connector to the structure, although not an activeone. It is now a failed persistent connector. In both cases, you can force thedeallocation of the structure as soon as it no longer has active connectors.

Allocated structures can be rebuilt. Rebuilding is the process of reconstructing astructure on the same or another CF. A rebuild consists of three main steps. First,XES allocates the new structure instance. Then, the data of the old structure isreconstructed in the new structure. Finally, XES deallocates the old structureinstance. Note that you cannot specify the target CF in your rebuild request. Aswith structure allocation, XES selects it from the preference list.

There are two methods for rebuild: user-managed and system-managed. Withuser-managed rebuild, the active connectors are responsible for reconstructing thedata. With system-managed rebuild, XES transfers the data to the new structureinstance. System-managed rebuild is thus also available for structures withoutactive connectors. These structures can either themselves be persistent or havefailed persistent connections.

When an application component connects to a structure, it specifies whether itallows the structure to be rebuilt through user-managed or system-managedrebuild. For structures with active connectors, both rebuild methods require that allactive connectors allow the respective rebuild method.

You can also duplex structures. Duplexing means maintaining two instances of thesame structure on different CFs at the same time. Duplexing serves to increaseavailability and usability of a structure.

Typical management tasks for CFs are removing a CF from the sysplex andreintegrating it again. These tasks have several steps that must be performed in acertain order and can be quite complex. To simplify these operations, SA z/OSoffers the INGCF command. INGCF has several functions, which serve tomanipulate structures and the CFs themselves. For more information, see IBMTivoli System Automation for z/OS Operator's Commands and the online help.

Some functions deal with the sender paths of a coupling facility. They have thefollowing limitations. First, at least one system in the sysplex that is running theautomation must know the control unit ID (CUID) of the coupling facility. If this isnot the case, no missing sender paths can be resolved.

A missing sender path occurs when a coupling facility is deactivated prior to asystem IPL (or reIPL) and then activated afterwards. The system that has been

Managing Coupling Facilities


IPLed (or reIPLed) does not recognize the coupling facility. To determine themissing sender paths, the automation calls the HOM interface of HCD. Resolvingthe missing path information is only possible when either the complete networkaddress is defined in HCD along with the processor ID, or you provide the CPCsynonym used by the automation as the processor ID. However, it is recommendedthat you define both. If neither is defined, the system that misses the sender pathsmust run the automation.

Recovery Actions

Resolving WTO(R) Buffer ShortagesWhen all WTO(R) buffers are in use, it is possible that commands can no longer beprocessed. To resolve this, there are several options: you can extend the buffer,change the properties of the affected consoles, or cancel jobs that issue WTO(R)s.

SA z/OS provides recovery of buffer shortage in two stages. It first tries to extendthe buffer and modify the console characteristics, if applicable. If this does nothelp, it then cancels jobs that issue WTO(R)s. You must specify which jobs can becanceled by SA z/OS if there is a buffer shortage.

Customization: Automation of buffer shortage recovery is enabled using theSA z/OS customization dialogs. For more information, see “Enabling WTO(R)Buffer Shortage Recovery” on page 148.

WTO Recovery is performed when different messages are received by SA z/OS.The action taken when each of these messages is received is described in Table 21on page 138.

Managing Coupling Facilities


Table 21. WTOBUF Recovery Process

Recovery Message Command

WTO IEA405E Set the console attributes.

If the deletion mode is not roll or wrap, set the mode to roll. K S,DEL=R,L=x

If any out-of-line display area exists, delete the status display. K E,D,L=x

If the interval between message rolls is greater than 1 second andnot '*' , set the interval to 0.25 seconds.

K S,RTME=1/4,L=x

If the console receives messages not only from the local systemand the WTO message buffer size has reached its maximum,remove the buffering systems from the list and add the localsystem to the list.

V CN(x),MSCOPE=(l)

IEA404A Suspend the console.

Requeue the messages to the hardcopy log. K Q,L=x

Vary the active console (COND=A) offline. For SMCS consoles,issue the appropriate VTAM command.

V {CN(x),OFFLINE|NET,TERM,LU1=x,TYPE=FORCE}

Cancel the job or TSO user that caused the shortage, but onlywhen defined as a candidate during customization.

C {jobnm,A=asid|U=userid}

IEA406I Resume the console if it was suspended and if it is not a SMCSconsole.

V CN(x),ONLINE

Restore the console attributes.

Set the deletion mode to the value before the buffer shortageoccurred.

K S,DEL=old,L=x

Set the interval between message rolls to the value before thebuffer shortage occurred.

K S,RTME=old,L=x

Set the list from which the console is to receive unsolicitedmessages to the list before the buffer shortage occurred.

V CN(x),MSCOPE=(l)

Increase the WTO message buffer size to minimize future shortagesas follows:

new = min(9999,max(1500,1.2 * current MLIM))

K M,MLIM=new

Issue the message AOF929 for permanent changes (MLIM).

WTOR IEA230E Increase the maximum number of reply IDs to the maximumallowable value if the maximum number of systems in the sysplexis greater than 8 or the system runs in local mode.

K M,RMAX=9999

Increase the WTOR message buffer size if the current RMAX valueis greater than the current RLIM value as follows:

new = min(9999,max(10 + 2 * maxsys_in_sysplex,1.2 * currentRLIM))

K M,RLIM=new

IEA231A Cancel all jobs and TSO users that have outstanding WTORs andthat are defined as candidates during the customization.

C {jobnm,A=asid|U=userid}

IEA232I Issue the message AOF928 for irreversible changes (RMAX). Issuethe message AOF929 for permanent changes (RLIM).

Handling Long-Running Enqueues (ENQs)This type of recovery is divided into the following individual functions:v Long-running enqueue recovery

Recovery Actions


v "Hung" command recoveryv Command flooding recovery

All these recoveries can be enabled and disabled individually or globally.

The long-running enqueue recovery function lets you:v Check which resources are blockedv Customize automation to cancel or keep the jobs that block the resourcev Customize automation to dump the jobs before they are canceled

You can determine which resources you want to monitor. You can define a valuefor the maximum time a job can lock a resource while other jobs are waiting for it.If this amount of time is exceeded, recovery takes place. Identification of andelimination of these potential bottlenecks helps to reduce the risk of a ParallelSysplex outage.

While the time definition describes an inclusion list, you also have the possibilityto define an exclusion list of resources that are not monitored at all.

For more information about enabling the ENQ function, see “Enabling LongRunning Enqueues (ENQs)” on page 151.

This function has been extended by two supplementary functions:v “"Hung" Command Recovery”v “Command Flooding Recovery” on page 140

"Hung" Command Recovery: The purpose of this function is to detect hungcommands that often result in multisystem outages. We distinguish threesituations:1. Commands that inhibit other commands from completing execution2. Commands that inhibit jobs from completing execution3. Jobs that inhibit commands from completing execution

Automation examines ENQ contention associated with command processing andbuilds a list of blockers and waiters. The SA z/OS policy is then examined to seehow long waiting commands and waiting jobs are allowed to wait beforeautomated action is taken. The policy is also examined to determine what action(DUMP, NODUMP, KEEP or exclude) is to be taken against the blocking commandor job, as follows:1. When a command inhibits other commands from completing and no policy

definitions exist for any of the waiting commands, no automated action istaken.

2. When a command inhibits jobs from completing and no policy definitions existfor the blocking command, no automated action is taken.

3. When a job inhibits commands from completing and no policy definitions existfor any of the waiting commands, no automated action is taken.

If long-running ENQ and hung command recovery detect that the same resourcerequires automated action at the same time, the hung command recovery policydefinitions take precedence and hung command recovery automates the resource.

The action taken (DUMP, NODUMP, KEEP or exclude) is identical to thelong-running ENQ recovery action.

Recovery Actions


In either case only commands that are waiting on blocked resources areconsidered. "Hung" command recovery only considers those resources that are notbeing monitored by long-running ENQ recovery. If long-running ENQ recovery isdisabled then all resources, even those defined as long-running ENQ resources, areconsidered for "hung" command recovery. It is also important to realize that iflong-running ENQ recovery is enabled and a generic "catchall" resource definitionapplies, then "hung" command recovery cannot occur, because long-running ENQrecovery always take precedence.

Commands are executed by the master and console address spaces. Thus when aresource blocker is from either of these address spaces it is considered to be ablocking command rather than a blocking job.

As with resources, you can make similar definitions for commands that determinehow long a command is permitted to lock a resource while other commands arewaiting for the resource.

If the resource blocker is a job then recovery actions are only taken when the jobhas blocked the command for 3 consecutive iterations of "hung" commandrecovery processing. This results in a job blocking a command for no more than 90to <120 seconds.

Recovery action for the blocking job or the job that issued the blocking commandis the same as that specified for long-running ENQ recovery automation.

Command Flooding Recovery: The purpose of this function is to detect jobs thatflood a command class. Command flooding can cause log buffer shortages andinhibits other commands from executing. Both can lead to a multisystem outage.

When all (50) TCBs that are reserved for command processing are in use, newcommands are queued to the waiting queue. In this case the system issues messageIEE806A which triggers this function to evaluate what jobs are causing thesituation.

Jobs that just issue a set of commands, such as 200 (or more) "VARY dev,ONLINE"commands should not be considered during the evaluation. This is achieved bycomparing the current and the previous snapshot of the affected command class.

Snapshot processing is scheduled when message IEE806A is trapped. The intervaltime between the snapshots is 3 seconds by default (see “Enabling Long RunningEnqueues (ENQs)” on page 151 for details about adjusting this value if necessary).The interval should give these jobs enough time to finish issuing commands beforethe first snapshot is taken. Only jobs that issue commands on two consecutivesnapshots become subject of the recovery action.

Before the recovery action takes place, the number of commands that are issued bythe job must exceed a threshold (see below) and at least one of the commandsmust not be involved in a lock contention that is handled by the "hung" commandsrecovery.

The recovery action depends on the job definitions (see “Enabling Long RunningEnqueues (ENQs)” on page 151). If the job can be canceled, the recovery alsoremoves its waiting commands and terminates its executing commands. Therecovery action is completed either with message ING922E or with messageING924E. The latter message is repeatedly issued approximately every minute untilthe waiting queue becomes empty.

Recovery Actions


The threshold is calculated by subtracting the number of jobs that are issuingcommands in the command class from the total number of TCBs (50) that arereserved for command processing. This prevents jobs that repeatedly issue fewcommands from being evaluated.

The recovery ends when the message IEE061I is issued.

Note: The dump definitions are not in effect if a dump should be taken when thejob is canceled. This is because the recovery routine of the job that is beingcanceled can suppress the dump.

Customization:Automation of handling long-running enqueues is enabled through the SA z/OScustomization dialogs. For more details, see “Enabling Long Running Enqueues(ENQs)” on page 151.

Managing System RemovalThe purpose of this function is to isolate failed systems from a Parallel Sysplex byremoving them as quickly as possible. It also ensures fast mean time to recovery(MTTR) for those system images that you wish to restart immediately if anunavoidable outage occurs.

Note: This function is unavailable when running on a z/OS image which runsunder z/VM, even if the function is enabled.

In particular, the function automates the messages IXC102A and IXC402D.

The automation of the IXC102A message completes the Sysplex FailureManagement (SFM). Under certain circumstances SFM cannot complete theisolation of a failed system. This is because SFM's HW isolation, resetting thechannel subsystem (CSS) of the failed system, is driven through the CF. Whenconnectivity between the system image and the coupling facility is lost, SFMcannot perform the hardware isolation (ISOLATE command) and defers resettingthe system image until manual operator intervention occurs. Message IXC102Atells the operator to manually reset the HW and then reply "DOWN" to themessage, after which SFM safely partitions the system image out of the sysplex.The longer the delay lasts, the more the components and applications that rely onXCF messaging are impacted. The delay can eventually lead to a sysplex outagewhen the failed system has I/O operations pending. Automation of this messageminimizes the delay.

Message IXC402D has the same impact as IXC102A. However, this messageindicates a possible temporary inoperative status of the system due to a missingstatus update. For this reason the automation gives the system the chance torecover before the removal takes place by replying "INTERVAL=sss" to the firstoccurrence of message IXC402D. The interval time, sss, is the failure detectioninterval that is displayed by the command D XCF,CPL.

The automation does the removal of a system in two stages. The first stage clearsany pending I/O operations by sending a hardware command to the SupportElement. This requires information about the software running on the hardware.Because the system issuing message IXC102A or IXC402D does not necessarilyhave access to the hardware of the failed system, the automation needs predefinedmapping between software and hardware. Depending on this mapping, it thenroutes the hardware command to the system that has access to the hardware of thefailed system. For information about how to do the mapping refer to “Enabling

Recovery Actions


System Removal” on page 148. For further information about the hardwarerequirements refer to IBM Tivoli System Automation for z/OS Planning andInstallation.

The second stage replies to the outstanding WTOR with "DOWN" triggering theremoval of the system from the sysplex.

Customization:Automation of message IXC102A is enabled through the SA z/OS customizationdialogs. For more details, see “Step 4: Automating IXC102A and IXC402DMessages” on page 149.

Recovering Auxiliary Storage ShortageWith the automation of local page data sets that is controlled by the recovery flagof the automation flag MVSESA.PAGE, you can prevent auxiliary storage shortageoutages by dynamically allocating spare local page data sets when needed.

The function checks which job causes the shortage condition and whetheradditional page data sets can be added. If this is not possible, the job that iscausing the shortage will be cancelled if this has been defined. For more details seeTable 22 on page 143.

To enable local page data set automation, you should customize the PAGTOTLparameter (defined in one of the IEASYSxx PARMLIB members used during IPL).Make sure that you set the PAGTOTL parameter to a value greater than thenumber of local page data sets currently used.

Local page data sets must be defined in the master catalog and should not beSMS-managed. It is recommended that you use pre-allocated local data sets insteadof dynamically allocated ones. This makes the process faster because formattingnewly allocated page data sets is time-consuming. Each predefined local page dataset should be allocated with 10% space of local page space that is currently usedby the system. If predefined page data sets can no longer be allocated, new localpage data sets will be created dynamically.

The following table shows the recovery actions in detail. Generally speaking;v In case the recovery cannot complete successfully it always terminates with

message AOF953 and a specific reason code.v On invocation due to page data set shortages the recovery first checks whether

some preformatted page data sets and the HLQ for data sets that should bedynamically created have been defined. If none has been defined the recoveryterminates with RSN=4.

v If the command PAGEADD returns a message other than IEE782I and IEE783Ithe recovery is terminated with RSN=24.

v If an IDCAMS service fails the recovery is terminated with RSN=16.

WorkLoad Manager (WLM) - responsible for the Auxiliary Storage Management(ASM) - recommends that you automate the message IRA210E rather than messageIRA206E which has replaced the message IRA204E since z/OS release 1.10. Thereason is that the system has already identified the address space that caused theshortage.

Storage class memory (SCM) available with zBC12 and zEC12 requires at leastz/OS release 1.13 with APAR OA38660.

Recovery Actions


||||

||

Table 22. Auxiliary Storage Management


MSG IRA205IIRA200EIRA201EIRA265IIRA260E

Check the maximum number of local page data sets. When thevalue PAGTOTL is reached the recovery is terminated RSN=20.

MVS D ASM,LOCAL

Add a free page data set that has been allocated and formattedin a prior recovery.

MVS PAGEADD

If the previous action was unsuccessful and message is notIRA201E create and add a page data set.

MVS PAGEADD

If the previous action was unsuccessful add a pre-formatted dataset.

MVS PAGEADD

If the previous action was unsuccessful and message is IRA201Ecreate and add a page data set.

MVS PAGEADD

If PAGEADD was successful schedule timer INGT742P foradding more page data sets in case this action was insufficient.Otherwise, the recovery is terminated with RSN=8 or RSN=12.

AFTER 00.00:10,ID=INGT742P

IRA203EIRA204EIRA210E

Save the jobname of the very first message only. Next, check ifthe job has been defined for cancellation. Finally, schedule atimer to pop up 10 seconds later than the timer INGT742P. If thistimer does not exist, use the current time.

AT hh:mm:ssID=INGT742J

IRA202IIRA262I

Delete any pending recovery action. MVS D ASM,LOCALPURGETIMER=INGT742PPURGETIMER=INGT742JPURGETIMER=INGT742S

IRA250I Determine whether SCM offline storage is available. If no morestorage can be added, the recovery is terminated with RSN=0,but will continue when IRA200E is issued.

MVS D M=SCM

Add 'increment size' of SCM storage. CF SCM(incr),ON

If CF SCM was successful schedule a timer for adding more SCMstorage in case this action was insufficient. Otherwise therecovery is terminated with RSN=28

AFTER 00:00:10,ID INGT742S

IRA252I Delete SCM pending recovery actions. PURGE TIMER=INGT742S

ILR009E Mark the page data set “unuseable” if it is a spare data set andremove the volume from the list of available volumes fordynamic allocation.

MVS D ASM,LOCAL

If the data set was a spare data set:

v Check the maximum number of local page data sets. When thevalue PAGTOTL is reached the recovery is terminated withRSN=20

v Create and add a page data set. If PAGEADD wasunsuccessful the recovery is terminated with RSN=8 orRSN=12

PAGEADD

IEE205I Mark the page data set ‘free’ if it is a spare data set.

PAGE n/a Repeats the recovery of messages IRA200E, IRA205I, IRA260Eand IRA265I after the timer INGT742P has expired.

PAGECRIT n/a Repeats the recovery of messages IRA201E after the timerINGT742P has expired.

SCM n/a Repeats the recovery of message IRA250I after the timerINGT742S has expired.

Recovery Actions


||||||

|||

|||

||||

||

||||||||||

||||

|

|||

|||||

|||||

Table 22. Auxiliary Storage Management (continued)


JOB n/a Cancels the job that caused the recovery after the timer INGT742Jhas expired.

MVS CANCEL

Hardware ValidationThis function performs cross-validation of the hardware configuration mapped outin the customization dialogs against the actual hardware configuration that isrunning. This information is critical to accurately control logical partitions (LPARs)on any supported CPC within the HMC/SE LAN over the BCP Internal Interface.

Hardware validation uses the CPC name, Partition name and Partition number toensure that the LPARs defined in the customization dialogs are on the correct CPCand located on the correct partition number. However, this helps only for couplingfacilities because their partition identifiers must be defined in the active CFRMpolicy.

For MVS images, information from the HMC/SE (such as system name andsysplex name that are stored during initialization) is used to verify thecorresponding customization dialog definitions. During initialization of theautomation's Hardware Command Interface and just before a disruptive request issent to a partition, new checks are made to ensure that everything matchescorrectly.

Note: Only active images can be verified. For inactive images we must still rely ondefinitions made in the customization dialogs.

An active system in this context is a system belonging to the same sysplex as thesystem that runs the hardware validation, that is SA z/OS checks only systemsand coupling facilities within its own sysplex.

Hardware validation runs on an SA z/OS system primarily during startup, andsubsequently when changes to the definition in the customization dialogs areapplied through the INGAMS REFRESH command. The validation checks thedefinitions of all registered systems, that is whenever an SA z/OS systemperforms the hardware validation, it validates all systems and coupling facilitiesthat are active in the sysplex at this point in time. Registered systems are systemsrunning SA z/OS that have joined the same XCF group.

The validation of active systems and coupling facilities requires that the CPCs thathost the active systems must all be defined in the customization dialogs.

The data for inactive systems cannot be verified. However, these definitions arechecked for consistency across all registered systems. As soon as one of theseinactive systems or coupling facilities joins the sysplex or is made available for use,the validation is run for the particular image only.

Retrieving actual hardware information can take up to 5 minutes per CPCdepending on the model and its LPARs. During the time that the hardwarevalidation takes place all other hardware-related automation is either delayed orcannot be performed, depending on the type of recovery. For this reason thevalidation carries out "delta" processing. That is validating only the data that haschanged. This also includes the absence of data resulting in terminating CPCconnections when CPC definitions are missing that have been applied by a prior

Recovery Actions


validation. The actions resulting from the validation are performed on ALLregistered systems. This has two advantages:v you don't need to recycle NetView for changes in hardware definitions.v you only need to make the changes available to one system.

The first part of the hardware validation triggered by the ACF command or theautomation startup determines what CPC connections must be terminated andinitiated, namely in this sequence. The resulting actions are performed on allregistered systems. When this step has been completed successfully the imagevalidation is performed.

The image validation collects actual hardware information, and verifies the currenthardware definitions against the actual data and the definitions found on all otherregistered systems. It informs you if:v A real system or coupling facility could not be validated because either actual

hardware information or user definitions are not availablev The image definitions could not be evaluated because the actual hardware

information is not availablev The real system or coupling facility is not active and the image definitions of

some of the registered systems are differentv Any definition value has been corrected that was improperly defined or not

defined at all

Changes in hardware definitions can be made available to all registered systems bysimply invoking the command INGAMS REFRESH on only one of the thesesystems. There is one exception: the change of the authorization token value usedfor the communication with a particular CPC. A change of this value requires 3steps:1. In the first step you must remove the particular CPC definition and then

invoke the ACF command as above.2. When the command completes successfully the next step is to change the

authorization token value of the CPC at the Support Element.3. The final step is to define the CPC again with the new token value and invoke

the ACF command again.

Note: This behavior of the INGAMS command applies to the hardware definitionsonly.

The second part of the validation is triggered by either the message IXC517I that isissued when a coupling facility is made available for use, or by the automationitself when notified that a system joined the sysplex. Both trigger the automationto perform only the validation of the new system or coupling facility. Multipleoccurrences of messages for the same system or coupling facility are ignored whilethis system or coupling facility is validated. In case of a new system, the advantagehere is that the real hardware is validated before the system starts NetView andthe automation. If this automation then detects no difference between its currentdefinitions and the definitions of the other registered systems—which is the normalcase—only a consistency check takes place. This check does not require any realhardware information.

Prerequisites

Note: Hardware validation is not supported on MVS systems running underz/VM.

Hardware Validation


Enabling Hardware-Related AutomationTo enable the sysplex automation that SA z/OS provides for recovery actions andcoupling facility management, the following definitions must be made in thecustomization dialog.

Step 1: Defining the ProcessorUse the customization dialog to define a new processor of Entry Type PRO. Thename should be the real physical name of the processor defined in HCD. For moreinformation, refer to the online help or the section "Creating a New Processor" inIBM Tivoli System Automation for z/OS Defining Automation Policy.

Step 2: Using the Policy Item PROCESSOR INFOUse the Processor Information panel, to define a processor using Entry Type PRO.

Note: The connection type protocol must be INTERNALFor more information, refer to the online help or the section "More about PolicyItem PROCESSOR INFO" in IBM Tivoli System Automation for z/OS DefiningAutomation Policy.

Step 3: Defining Logical PartitionsIf the processor that you have defined runs in LPAR mode, define its logicalpartitions using the LPAR Definitions panel. You should define all LPARs that arephysically available on your processor, together with the systems that run on them.

For more information, refer to the online help or the section "More about PolicyItem LPARS AND SYSTEMS" in IBM Tivoli System Automation for z/OS DefiningAutomation Policy.

Step 4: Defining the SystemDefine a system using entry type SYS, and the Define New Entry panel.

Note: To avoid receiving hardware validation messages during SA z/OSinitialization, you should define all your systems (including your couplingfacilities).For more information, refer to the online help or the section "Creating a NewSystem" in IBM Tivoli System Automation for z/OS Defining Automation Policy.

Step 5: Connecting the System to the ProcessorConnect this system to the processor that you defined in “Step 2: Using the PolicyItem PROCESSOR INFO” and to its logical partition (if you set the processor modeas LPAR).

Connect this system to the sysplex or standard group (see “Step 6: DefiningLogical Sysplexes” on page 147 and “Step 7: Defining the Physical Sysplex” onpage 147).

Note: MVS SYSNAME and the Image/ProcOps Name must be the same.

Restriction: Usually, the MVS SYSNAME may begin with a number. However, inthis case, it must be the same as the Image/ProcOps Name, which cannot beginwith a number. Therefore, this naming restriction also applies to the MVSSYSNAME.

Enabling Hardware-Related Automation


Step 6: Defining Logical SysplexesDefine EACH logical sysplex (systems within the same XCF group ID) using entrytype GRP with group type SYSPLEX.

Use policy SYSPLEX to enter the real physical sysplex name. You can use the samename in several SYSPLEX GRPs.

Use policy SYSTEMS to connect all systems within the same XCF group ID to theSYSPLEX GRP. A system can only be connected to one SYSPLEX GRP.

Step 7: Defining the Physical SysplexDefine your real physical sysplex using entry type GRP with group typeSTANDARD.

Use policy SYSTEMS to connect all systems of your physical sysplex to theSTANDARD GRP.

Enabling Continuous Availability of Couple Data SetsCouple data sets (CDSs) contain important information about how to managecertain aspects of your sysplex. For example, the SFM CDS (sysplex failuremanagement couple data set) defines how the system manages system andsignalling connectivity failures and PR/SM™ (Processor Resource/SystemsManager™) reconfiguration actions.

The following couple data sets are particularly important for the functioning ofyour Parallel Sysplex:v The SYSPLEX couple data set, which defines the systems and the XCF groups of

the sysplexv The CFRM couple data set, which defines the coupling facilities and structures

of the sysplex

It is recommended that you define alternate couple data sets for all couple datasets in your sysplex. These alternate couple data sets serve as backups when theprimary CDS fails.

With the customization dialog you can specify a series of spare volumes for everyCDS type, for example, SYSPLEX, ARM, CFRM. The first volume in the series isused to create an alternative CDS if one of the primary alternate CDSs fails.

In the customization dialog you define the potential alternate couple data setsusing the Group entry type. Select a sysplex group, then select its policy itemSYSPLEX (define sysplex policy) from the panel Policy Selection.

The Sysplex Policy Definition panel is displayed if you select policy item SYSPLEXfrom the Policy Selection panel for sysplex groups.

For a description of this panel refer to the online help or the section "More AboutPolicy Item SYSPLEX" in IBM Tivoli System Automation for z/OS Defining AutomationPolicy.

Enabling Hardware-Related Automation


Enabling WTO(R) Buffer Shortage RecoveryYou can customize the WTO(R) buffer shortage recovery of SA z/OS with theMESSAGES/USER DATA policy item of the customization dialog for the MVSComponent entry type (MVC).

Code definitions for the message ID WTOBUF are used to specify jobs that arecanceled or kept in case a WTO(R) buffer shortage is threatening. The jobs that youselect for cancelation will then no longer issue WTO(R)s.

Specify code definitions for message ID WTOBUF with the following values:

CODE1Specifies the name of the job which might or might not be canceled.

CODE2This must be WTO, WTOR, or * to indicate which requests the job (or jobs)might or might not be canceled for. Use just * to specify WTO and WTORrequests.

CODE3This must be blank.

Value ReturnedThis must be CANCEL to indicate that the job (or jobs) might be canceledor KEEP to indicate that they might not.

Example:

Code 1 Code 2 Code 3 Value ReturnedJOB1 WTOR KEEPJOB2* WTO KEEPJOB3* * CANCELJOB4* * KEEP* * KEEP

To set up the default behavior for all jobs not explicitly defined, a specification ofCODE1=* and CODE2=* is needed. To indicate that all other jobs might becanceled specify CANCEL in the Value Returned field, otherwise specify KEEP.

The job name *MASTER* cannot be entered in the Code 1 field. Even if yourdefault behavior is set up to cancel all jobs that have not been explicitly defined, acancel command is not issued against *MASTER* if it is the job name beingchecked. This is because *MASTER* is non-cancelable.

Enabling System RemovalSA z/OS helps you to resolve pending I/Os for systems being removed from thesysplex.

See “Recovery Actions” on page 137 and “Managing System Removal” on page141 for further details.

Because the automation must know where the system is located to send thecommand to the appropriate Support Element, you must use the customizationdialog to define its hardware configuration.

Enabling WTO(R) Buffer Shortage Recovery


The BCP Internal Interface allows you to perform hardware operations from eachNetView in your sysplex member as long as its processor hardware supports this.Refer to IBM Tivoli System Automation for z/OS Planning and Installation for moreinformation.

Hint: If you want to use the IXC102A automation, make sure there is no processoroperation related IXC102A automation defined in your automation policy.Likewise, if you want to continue to use the processor operations basedautomation of messages IXC102A and IXC402D, the IXC102A automation flag mustbe disabled.

Step 1: Defining the Processor and SystemThe processor and system must be defined as described in “EnablingHardware-Related Automation” on page 146.

Step 2: Defining the Application with Application Type IMAGEUse entry type APL to define a new application with Application Type IMAGE andsubsystem name that is the same as the Image Name of the system that thisapplication represents (as defined in “Step 4: Defining the System” on page 146).

Use entry type APL and select policy item APPLICATION INFO for your system.On the panel Application Information you can define a new application type IMAGE.For more information, refer to the online help or the section "Policy Items forApplications" in IBM Tivoli System Automation for z/OS Defining Automation Policy.

Because the application has been defined as type IMAGE, the job name is set bydefault to the subsystem name and cannot be changed.

The Subtype, Scheduling Subsystem, JCL Procedure Name, ARM Element Name,and WLM Resource Name are forced to be blank.

Some other definitions in the policy item APPLICATION INFO are also defaulted:v the Job Type is forced to NONMVSv the Monitor Routine is defaulted to INGMTSYS if nothing is specifiedv the External Startup is defaulted to ALWAYS if the Monitor Routine is

INGMTSYSv the External Shutdown is defaulted to ALWAYS if the Monitor Routine is

INGMTSYS

For more information, refer to the online help or the section "More About PolicyItem APPLICATION INFO" in IBM Tivoli System Automation for z/OS DefiningAutomation Policy.

Step 3: Defining an Application GroupBecause you can only automate applications by linking them to systems via anapplication group, you need to define an application group for the IMAGEapplications.

Step 4: Automating IXC102A and IXC402D MessagesYou can automate IXC102A and IXC402D messages to avoid sysplex outages.

Note: The following shows examples for defining commands and codes formessage IXC102A.

Enabling System Removal


You can specify one of the following four hardware commands for each system inthe sysplex that is automated. If you specify nothing SYSRESET CLEAR is used.v SYSRESET [CLEAR]v DEACTIVATEv ACTIVATE [P(image_profile_name)]v LOAD [P(load_profile_name)]

Where:

CLEARIndicates that the storage is cleared

P Specifies the profile to be used. The name can consist of up to 16alphanumeric characters. If the parameter is omitted, the last used profileis taken.

Note: The following restriction applies to the hardware commands ACTIVATEand LOAD:

Both commands invoke processor functions that can cause asynchronous eventssuch as operator messages at BCP (Basic Control Program) Internal Interfaceinitialization time or processor hardware wait states. Currently, the BCP InternalInterface does not allow the monitoring and control of these events.

Use policy item MESSAGES/USER DATA of the SA z/OS customization dialogwithin the application type IMAGE you created to define commands and codes formessage IXC102A and IXC402D. Enter C or S in the Cmd column and IXC102A inthe Message ID column (or IXC402D for IXC402D message automation). For moreinformation, refer to the online help or the section "MESSAGES/USER DATAPolicy Item for Applications " in IBM Tivoli System Automation for z/OS DefiningAutomation Policy. The definitions here also apply to message IXC402D.

Pressing Enter displays the CMD Processing panel, as shown in Figure 27. Use thispanel to specify a valid hardware command for the image in the Command Textcolumn and a "Pass/Selection" value that must match the "Value Returned"definition specified on the Code Processing panel.

On the Message Processing panel enter k to define codes. Specify on the CodeProcessing panel, as shown in Figure 28, the following:

If you want to automate messages IXC102A and IXC402D , you must enter IXC102Afor Code 1 and BCPII for Code 2 for both messages.

Cmd Ps AutoFn/* Command TextACTCODE LOAD P(LOADPROF)

Figure 27. Sample Panel for Command Processing

Code 1 Code 2 Code 3 Value ReturnedIXC102A BCPII ACTCODE

Figure 28. Sample Panel for Code Processing



Step 5: Verify Automation table entriesVerify that the entries of IXC102A and IXC402D of the predefined messages areused in your automation table and that the auto-operator AUTXCF and AUTXCF2are defined (see *BASE sample policy).

Enabling Long Running Enqueues (ENQs)If you automate long running ENQs, you must define the following:v The resource or resources that are being checkedv The time frame when a long ENQ is detected

If you automate "hung" commands, you must define the following:v The command (or commands) that are being monitored or excluded from

monitoringv The time frame for each command that a command is granted for completion or,

if commands are to be excluded from monitoring, the exclusion keywordv The action to be taken against this command if this command is determined to

be a blocker of other commands or jobs

In addition, the following definitions can be made:v The names of jobs that should be canceled or kept when detecting a long ENQ,

a "hung" command, or command floodingv The snapshot interval for a command classv The title of the dump taken before the job is cancelledv The default storage areas to be dumpedv Symbol definitions to be used when the dump specifications are provided by a

PARMLIB member

Use the entry type GRP in the customization dialog to define the followingpolicies:v Resource definitionv JOB/ASID definitionsv IEADMCxx symbolsv Command definitionv Snapshot interval definition

Step 1: Defining ResourcesUse the Long Running ENQ Resource Definition panel to define your resources.This panel is displayed if you select policy item RESOURCE DEFINITIONS fromthe Long Running Enqueue Policy section of the Policy Selection panel for sysplexgroups. For more information, refer to the online help or the section "More AboutPolicy Item RESOURCE DEFINITIONS" in IBM Tivoli System Automation for z/OSDefining Automation Policy.

Step 2: Making Job/ASID DefinitionsUse the Long Running ENQ Job/ASID Definitions panel that is displayed if youselect policy item JOB/ASID DEFINITIONS from the Long Running EnqueuePolicy section of the Policy Selection panel for sysplex groups. For moreinformation, refer to the online help or the section "More About Policy ItemJOB/ASID DEFINITIONS" in IBM Tivoli System Automation for z/OS DefiningAutomation Policy.



Step 3: Defining IEADMCxx SymbolsUse the Long Running ENQ IEADMCxx Symbols panel that is displayed if you selectpolicy item IEADMCxx SYMBOLS from the Long Running Enqueue Policy sectionof the Policy Selection panel for sysplex groups. For more information, refer to theonline help or the section "More About Policy Item IEADMCxx SYMBOLS" in IBMTivoli System Automation for z/OS Defining Automation Policy.

Step 4: Defining CommandsUse the Long Running Command Definition panel to define your commands. Thispanel is displayed if you select policy item COMMAND DEFINITIONS from theLong Running Enqueue Policy section of the Policy Selection panel for sysplexgroups. For more information, refer to the online help or the section "More AboutPolicy Item COMMAND DEFINITIONS" in IBM Tivoli System Automation for z/OSDefining Automation Policy.

Step 5: Defining Snapshot IntervalsUse the Command Flooding Definition panel to define the individual snapshottimes. This panel is displayed if you select policy item COMMAND FLOODINGfrom the Long Running Enqueue Policy section of the Policy Selection panel forsysplex groups. For more information, refer to the online help or the section "MoreAbout Policy Item COMMAND FLOODING" in IBM Tivoli System Automation forz/OS Defining Automation Policy.

Enabling Auxiliary Storage Shortage RecoveryTo prevent auxiliary storage shortage outages you can predefine local page datasets, using the SA z/OS customization dialog for entry type GRP to define thefollowing:v local page data setv job definitions

Step 1: Defining the Local Page Data SetUse the Local Page Data Set Recovery panel that is displayed if you select policyitem LOCAL PAGE DATA SET from the Local Page Data Set Policy section of thePolicy Selection panel for sysplex groups.

For more information, refer to the online help or the section "More About PolicyItem LOCAL PAGE DATA SET" in IBM Tivoli System Automation for z/OS DefiningAutomation Policy.

Step 2: Defining the Handling of JobsUse the Local Page Data Set Recovery Job Definition panel that is displayed if youselect policy item JOB DEFINITIONS from the Local Page Data Set Policy sectionof the Policy Selection panel for sysplex groups.

For more information, refer to the online help or the section "More About PolicyItem JOB DEFINITIONS" in IBM Tivoli System Automation for z/OS DefiningAutomation Policy.

Enabling Long Running Enqueues (ENQs)


Defining Common Automation ItemsThere are definitions that relate to utilities running as a started task. The first one(Temporary Data Set HLQ/TEMPHLQ) replaces the usage of the first qualifier ofthe automation status file.

The second definition (Started Task Job Name/STCJOBNM) allows the uniqueassignment of started task job names scheduled by the automation in case youhave dedicated job name assignments that conflict with the procedure namesprovided by the automation.

It is recommended that you define the Temporary Data Set HLQ/TEMPHLQ. If itis not defined, the automation uses the first qualifier of the automation status file.

You can define both of these items using the Sysplex Policy Definition panel that isdisplayed if you select the policy item SYSPLEX from the Policy Selection panel forsysplexes. For more information, refer to the online help or the section "MoreAbout Policy Item SYSPLEX" in IBM Tivoli System Automation for z/OS DefiningAutomation Policy.

Customizing the System to Use the Functions

Additional Automation Operator IDsTo automate the Parallel Sysplex, you must define the additional automationoperator IDs.

You can import these auto-operator definitions from the *BASE sample policyprovided.

Switching Sysplex Functions On and OffUse the SA z/OS customization dialog to specify the following minor resourcenames:

CDS For the recovery of alternate CDSs.

ENQ Enables the handling of the next four individual recoveries.

ENQ.CMDFLOODEnables the handling of commands that flood a particular command class.

ENQ.HUNGCMDEnables the handling of jobs and commands that inhibit other commandsfrom completing execution.

ENQ.LONGENQEnables the handling of long-running ENQs.

LOG For the recovery of the system log.

LOGGERFor the recovery of the system logger.

PAGE For the recovery of auxiliary storage shortage.

WTO For the recovery of WTO(R) buffer shortages.

XCF For automating messages IXC102A and IXC402D.

Defining Common Automation Items


By default, all recovery actions are enabled. If you want to disable them, use thecustomization dialog Flag Automation Specification and set the recovery flag to NO.

Note: You can change the automation recovery flag during run time by using thecommand INGAUTO.

Customizing the System to Use the Functions


Chapter 13. Automating Networks

Automation Network Definition ProcessThis section summarizes the steps for defining an automation network toSA z/OS. More detail for each step of the process is provided later in thisinformation.1. Define your message forwarding paths between different systems. To do this,

you define:v A primary focal point, where all notifications are sent.v An optional backup focal point, used when the primary focal point is

unavailable.v Target systems, which are monitored and controlled by the focal point

system.v Gateway sessions between the systems.

“Defining Gateway Sessions” on page 156 describes how to define gatewaysessions.

2. Modify the NetView definitions to reflect your automation networkconfiguration. The chapter on how to install SA z/OS on host systems in IBMTivoli System Automation for z/OS Planning and Installation provides details.

For an example of the automation network definition process, see also the chapterabout installing SA z/OS on host systems in IBM Tivoli System Automation for z/OSPlanning and Installation.

These definitions create a path allowing message forwarding from target systemsto the focal point system.

A message forwarding path is best implemented by defining systems in thefollowing top-down manner:1. Primary focal point system2. Backup focal point system3. Target systems

Defining the primary focal point first ensures that it is ready to handle forwardedmessages as soon as forwarding is turned on for the target systems.

If the message forwarding path is not yet implemented on all systems in anautomation network, messages are displayed to notification operators on the targetsystems. Once the message forwarding path is implemented, notifications areforwarded to the focal point system.

If the target systems are implemented first, additional overhead occurs because thetarget systems unsuccessfully attempt to forward notifications, and the notificationsare logged in the NetView log.

Defining an SDF Focal Point SystemThe focal point system and backup focal point systems are defined using theNetwork entry type in the customization dialog. Each system has a single entry inthe automation policy defining the next system or domain in the message


forwarding path. Figure 29 shows an example automation network. In thisexample, the primary focal point system is CHI01. The backup focal point isCHI02.

SA z/OS always displays messages on the local system. It forwards them to afocal point, if there is one defined.

Defining Gateway SessionsTo define gateway sessions:1. For each system, define the outbound gateway autotask (GATOPER) on the

Automation Operators policy object of the customization dialog. Refer to"Automation Operators Entry Type" in IBM Tivoli System Automation for z/OSDefining Automation Policy.

2. On the SA z/OS Network policy object, use the GATEWAY policy item todefine the domains of all systems within the network which need tocommunicate with each other. Refer to "Network Entry Type" in IBM TivoliSystem Automation for z/OS Defining Automation Policy.

3. Define operator IDs used for all inbound and outbound gateway autotasksused on the system in the NetView DSIPARM data set member DSIOPF. See thechapter on how to install SA z/OS on host systems in IBM Tivoli SystemAutomation for z/OS Planning and Installation for details.

Defining Automatically-Initiated TAF Fullscreen SessionsYou can automatically establish Terminal Access Facility (TAF) fullscreen sessionsfor applications that SA z/OS monitors, so that the operators need not define thesessions on a daily basis.

These TAF fullscreen sessions are defined in the FULL SESSIONS policy item for aNetwork policy object.

In addition to defining TAF fullscreen sessions using the customization dialog, youfollow the NetView process for customizing TAF fullscreen sessions, as outlined inTivoli NetView for z/OS Customization Guide.

Once TAF fullscreen sessions are set up, they can be managed using the TAFFullscreen Menu in the SA z/OS Operator Interface. See IBM Tivoli SystemAutomation for z/OS User's Guide for more information on managing TAF fullscreensessions.

Primary focal point

Target system

Target system

CHI01 CHI02

ATL01

ATL02

Backup focal point

Figure 29. Focal Point Forwarding Definitions for Systems

Automation Network Definition Process


||||

||||

To define an application on the Fullscreen TAF Application Definition panel thatyou reach by selecting the FULL SESSIONS policy item of a Network policy object,specify the following:v The session name, or the name of the application that a TAF fullscreen session is

to be established for, for example, TSO. This name is displayed in theDescription field on the TAF Fullscreen Menu operator panel. This value can bethe same as that used for the application ID.

v The application ID. You can obtain this value from the library containing thenetwork definitions (VTAMLST) or from your network system programmer.

v The system name that the application runs on, for example, CHI01. This is aninformation-only field and is displayed in the System field on the TAFFullscreen Menu operator panel.

For example, the following panel defines a TAF fullscreen session for TSO insystem CHI01:

COMMANDS HELP------------------------------------------------------------------------------AOFPINE3 Fullscreen TAF Application Definition Row 1 to 10 of 20Command ===> SCROLL===> PAGE

Entry Type : Network PolicyDB Name : USER_PDBEntry Name : FOCAL_NETWORK Enterprise Name : USER_ENTERPRISE

Enter the applications with which SA z/OS operators can establish TAFsessions automatically using the operator interface.

Session Name Application ID SystemTSO TAIN1 CHI01

Figure 30. Fullscreen TAF Application Definition Panel

Defining Automatically-Initiated TAF Fullscreen Sessions

Chapter 13. Automating Networks 157

Chapter 14. Defining a VTAM Application to SA z/OS

VTAM applications need to have Application nodes activated for the application tooperate correctly. This is normally not a problem if an application is to run on asingle system. However, if the application is to be switched from one system toanother (via a move group or server group), the application node definition mustbe deleted from the system that the application is moving from. If this is not done,users may not be able to log on to the application because there is a definition forthe application node that is not active, that is, the application has not opened thenode ACB.

To alleviate this problem, the application node must be deleted from the oldsystem and created on the new system. Unfortunately, the only way to delete anode in VTAM is to deactivate its major node, that is, the member that it is definedin.

Newer releases of VTAM have introduced the concept of Model node definitions. Inthis case a major node is created with the type of node and a name that includeswildcards. Whenever a node of the type is accessed, VTAM will use the namerequested to match the models. It will then dynamically create a node based on themodel definitions with the name requested. When the node is no longer required itwill delete it. What this means for application nodes is that a model definition canbe defined once on each VTAM in the network that the application might be runon. Then when the application is started and opens its ACB, VTAM willdynamically create the node for it. Likewise when moving the application, uponclosing the ACB, VTAM will delete the node and another VTAM on anothersystem will dynamically create the node.

VTAM applications may require recovery commands to be issued if VTAM isrestarted, or the VTAM application node is reactivated. These commands differfrom subsystem to subsystem and can be specified in the Messages/User Datapolicy items as described in the following section.

The INGVTAM command provides a method of activating the Major Nodes for anapplication before the application is started, and deactivating the Major Nodesafter it is down. To enable the function you must code the INGVTAM command inthe PRESTART, REFRESHSTART, and SHUTDOWN FINAL definitions. In addition,if VTAM should ever be restarted whilst the applications are running, the majornodes must be reactivated. This can be accomplished by coding the INGVTAMcommand in the UP messages/user data policy for the VTAM subsystem.

Registering VTAM Application Subsystems with SA z/OS RecoveryTo enable VTAM application recovery to take place, the subsystems must beregistered with the SA z/OS recovery code.

This is achieved by using the INGVTAM command that is described in IBM TivoliSystem Automation for z/OS Programmer's Reference. The following application policyitems must be customized:1. The PRESTART policy in the STARTUP policy must have at least a NORMal

start item with the INGVTAM command to activate a list of major nodes. Thefollowing command can be used as an example:INGVTAM &SUBSAPPL REQ=ACTIVATE MAJNODE=(majnode1,majnode2,...)


|||||||

|||

Where majnoden are VTAM application major nodes. Each major mode is variedactive to VTAM when the subsystem prestart commands are issued. Note, it isexpected that only one of the major nodes contain the minor node that theVTAM application subsystem will use.

2. The FINAL definition in the SHUTDOWN policy is used to deregister thesubsystem with SA z/OS VTAM application recovery. Use the INGVTAMREQ=DEACTIVATE command in the policy. For example,INGVTAM &SUBSAPPL REQ=DEACTIVATE

The DEACTIVATE request issues a vary net inactive for each major noderegistered by the REQ=ACTIVATE. The vary is not done if the major node isshared by other subsystems that have also registered the major nodes. Whenthe last subsystem registered issues an INGVTAM REQ=DEACTIVATE, themajor node is varied inactive. The only exception to this is when the majornode contains model resources with wildcards in the node definition. In thiscase, the major node is never inactivated.

3. The REFRESHSTART definition must have the same definition as thePRESTART definition in the STARTUP policy. This policy item is used toreregister the subsystem with SA z/OS VTAM application recovery.

4. Enter commands that are issued when the VTAM subsystem is restarted.Typically, these commands reopen the VTAM ACB that the subsystem uses tocommunicate with VTAM.

5. Optionally enter a relationship for the subsystem to ensure that the prestartcommands are only issued when VTAM is up. The required relationship is:PrepAvailable(WhenAvailable),Passive,Weak -> VTAM/APL/=

where VTAM is the name of the VTAM subsystem and that is the supportingresource. Passive forces the relationship to wait until VTAM is UP. Weakspecifies that only the supporting resource status is checked.

6. In addition the UP message for VTAM should have the following command:INGVTAM REQ=ACTIVATE

When INGVTAM is executed with REQ=ACTIVATE and no positional subsystem,it finds all the subsystems that had previously registered via INGVTAM and issuesVary NET ACT commands for their major nodes. After this has been done, it willexecute any policy command(s) that is/are specified to USER MESSAGE VTAMUPfor the subsystems.

Defining a VTAM Application to SAz/OS


|||

|||

Chapter 15. Shutting Down z/OS systems in a GDPSEnvironment

SA z/OS allows you to shutdown z/OS systems either through the INGREQ ALLcommand in a GDPS® production environment or from a GDPS controlling system.

There are three distinct phases in the final shutdown processing that are definedusing the special message id SYSTEM_SHUTDOWN message/user data policyitem for the MVS Component entry type:

Phase 0This phase is entered prior to shutting down the resource that is identifiedby the GDPS STOPAPPL parameter (the STOPAPPL resource). In this phaseyou can perform any action before the actual system shutdown starts.

Phase 1This phase begins when the resource that is identified by the GDPSSTOPAPPL parameter (the STOPAPPL resource) has been terminated. Inthis phase you can specify additional INGREQ stop commands or anyother commands through NetView to terminate any subsystems that arestill present.

Phase 2This phase begins after the termination of OMVS and any local automationmanager (PAM and SAMs). Only NetView commands or z/OS commandsissued through NetView can be specified. For example, the MVS Z EODcommand.

Notes:

1. OMVS and all local automation managers are always shutdown bySA z/OS automatically. Do not specify termination commands forOMVS or automation managers in PHASE1 or PHASE2. When runningin a JES3 environment, consider triggering the command MVS FBPXOINIT,SHUTDOWN=FORKS via routine INGRYSHU in the JES3SHUTINIT phase. INGRYSHU will only issue the command when asystem shutdown is detected. Likewise, you can trigger command MVSF BPXOINIT,SHUTDOWN=FILESYS in the JES3 SHUTFINAL phase.For more information about INGRYSHU, refer to the description of theroutine in Automation Solutions.

2. Be aware that the NetView address space is still present and must stayup in order to signal the nearly termination of the system to GDPS.

3. Whenever NetView is ended without its CLOSE command beinginvoked, it cannot automatically archive recent messages in its activeCanzlog. This can be accomplished instead by invoking the NetViewCANZLOG CUE command during Phase 2.

The scenario is based on the provided best practice policies *BASE and *GDPS. Formore details refer to the MVC entry GDPS_SYSTEM_SHUTDOWN in the *GDPSbest practice policy.


||||||||

||

Example DefinitionThe actions you take to shutdown z/OS systems from within GDPS are definedusing the SYSTEM_SHUTDOWN message/user data policy item for the MVSComponent entry type. These actions can include instructing SA z/OS toshutdown resources out of the affected dependency path of GDPS STOPAPPL,shutdown file systems, and so on.

Table 23. Example SYSTEM_SHUTDOWN Command Processing Entry

Cmd Ps/Select AutoFn/* Command Text

PHASE1 INGREQ RACF/APL/&SYSNAME. REQ=STOP PRECHECK=NO REMOVE=SYSGONEVERIFY=NO OUTMODE=LINE

PHASE1 INGREQ RRS/APL/&SYSNAME. REQ=STOP PRECHECK=NO REMOVE=SYSGONEVERIFY=NO OUTMODE=LINE

PHASE1 INGREQ GDPS_ALL/APG/&SYSNAME. REQ=STOP PRECHECK=NO REMOVE=SYSGONEVERIFY=NO OUTMODE=LINE

PHASE1 INGREQ LOOKASIDE/APG/&SYSNAME. REQ=STOP PRECHECK=NO REMOVE=SYSGONEVERIFY=NO OUTMODE=LINE

PHASE1 INGRCHCK BASE_SYS/APG/&SYSNAME.OBSERVED=(SO HA)

PHASE1 INGRCHCK LOOKASIDE/APG/&SYSNAME.OBSERVED=(SO HA)

PHASE2 MVS Z EOD

Table 23 shows example definitions for the SYSTEM_SHUTDOWN entry that placestop votes against the listed resources in PHASE1 in a sequential order. Thedesired completion of the resource shutdown is processed in parallel. The specifiedINGRCHCK command at the end of the PHASE1 sequences waits for thecompletion of the stop requests for the specified resources.

For example:INGRCHCK LOOKASIDE/APG/&SYSNAME OBSERVED=(SOFTDOWN HARDDOWN)

For more information about the INGRCHCK command, see IBM Tivoli SystemAutomation for z/OS Programmer's Reference.

If synchronization is necessary, the FDBK option for the INGREQ commandpermits waiting until the appropriate subsystem has been shutdown. TheFDBK=WAIT option causes an INGREQ stop command to be processedsequentially. In this way it slows down the shutdown process.

The primary and all secondary automation managers (PAM and SAMs) on thelocal system will be shutdown by SA z/OS automatically unless they are movedto another system. OMVS will be shutdown by SA z/OS automatically too. Onlythe MVS Z EOD command is issued in PHASE2.

INGREQ Considerations

With GDPS V3.12 or below, the OVERRIDE and TYPE definitions used are thosedefined in the OVERRIDE definition of the GEOPLEX OPTIONS User E-T Pairs(UET) definition.

Shutting Down z/OS systems in a GDPS Environment


If nothing has been specified, a default of OVERRIDE=(TRG FLG) andTYPE=NORM is used.

For GDPS V3.13 or higher, the OVERRIDE definition in the GEOPLEX OPTIONS isno longer supported. Therefore, the default OVERRIDE=(TRG FLG) andTYPE=NORM is used and can be changed only via the INGREQ user exitAOFEXC01.

Shutting Down z/OS systems in a GDPS Environment

Chapter 15. Shutting Down z/OS systems in a GDPS Environment 163

Chapter 16. WTOR Processing

When System Automation for z/OS receives WTORs (write-to-operator-with-replyrequests), it either automatically replies to them, or stores them if they are to beused for recovery or to shut down the subsystem that issued them. WTORs thatare stored for later use are known as outstanding WTORs.

Process Flow of WTORsAll WTORs that are issued at a system should be forwarded to NetView.Otherwise SA z/OS is not able to process them.

To achieve this, specify YES for Route all WTORs to AT in policy BUILDCONTROL of Entry Type MSG. For details of this Entry Type and policy, refer toIBM Tivoli System Automation for z/OS Defining Automation Policy. Otherwise thefollowing definition in the message revision table has to be added:UPON (ALWAYS)

SELECT* Ensure all WTORs are being automated

WHEN (WQE SUBSTR 345 C2D ^= "+0")REVISE("Y" AUTOMATE)

OTHERWISEEND

Incoming WTORs are processed by the NetView automation table (AT) and thistriggers commands according to the processing purpose:

Called automation routine Processing Purpose

ISSUEACT 1. Issue commands or replies (or both) that have beendefined to a subsystem.

2. Store the WTOR if it has not been replied to.

ACTIVMSG, HALTMSG,TERMMSG

1. Update the status of the subsystem that issued theWTOR.

2. Issue defined commands or replies (or both).


INGMON 1. Issue commands or replies (or both) that have beendefined to a monitor resource.


OUTREP Store the WTOR.

The commands (other than OUTREP) are routed to the first active task that isdefined in the AT synonym %AOFOPGSSOPER%. Thus they are usually routed tothe work operator of the subsystem that issued the WTOR. This is done based onthe job name that is associated with the WTOR.

Automation routines that process WTORs from subsystems that are not defined inSA z/OS or are from MVS components are routed to tasks that the WTORs havebeen assigned to based on their message ID.

The OUTREP command is routed to the first active task that is defined in the ATsynonym %AOFOPSYSOPER%.


||||

Actions in Response to Incoming WTORsYou can use the MESSAGES/USER DATA automation policy item to define whatresponse SA z/OS should make to incoming WTORs for applications, monitorresources and MVS components, as follows:v Use the CMD action (possibly combined with the CODE action) to define

commands that are to be issued in response to an incoming WTOR.v Use the REP action (possibly combined with the CODE action) to define a reply

that is to be made immediately in response to an incoming WTOR.v Use the AUTO action to define the incoming WTOR as a status message that

changes the status of the subsystem that issued the WTOR.

NetView automation table statements are created that call the relevant command,depending on the defined actions.

If you used CODE definitions to define actions, the automation table statementsthat are created have to be supplemented with an OVR action to tell SA z/OSwhat variable information is to be extracted from the WTOR and how to pass thisdata as code values to the related command.

WTORs that have no actions defined for them are stored by SA z/OS viaOUTREP. Appropriate automation table statements are created for this purpose.

Customizing how WTORs Are Stored by SA z/OSSA z/OS keeps track of all outstanding WTORs that have not yet been replied toand displays them via SDF or NMC.

These outstanding WTORs include:v Permanent outstanding WTORs that are issued by applications at startup and

thus provide an interface for critical operator communication and shutdownv WTORs that no replies have been defined for in the SA z/OS automation policyv WTORs that were issued before SA z/OS had initialized or during down time

of SA z/OS

You can use the automation policy to define the severity for outstanding WTORsand a priority that allows you to distinguish between primary and secondaryWTORs:

SeverityThe severity of a WTOR determines the color of the WTOR in SDF andNMC. The following values can be specified for the severity:

NORMALOrdinary messages that do not indicate a problem.

UNUSUALMessages that might indicate a problem.

IMPORTANTMessages that indicate serious problems.

CRITICALMessages that indicate alarming problems.

IGNOREMessages that are to be ignored by SA z/OS.

WTOR Processing


||

PriorityA primary WTOR is stored and can later be used for operatorcommunication and to shut down the subsystem that issued it. In contrast,secondary WTORs are replied to immediately, or may be stored to bedisplayed in SDF and NMC.

This customization is done with code definitions in the MESSAGES/USER DATApolicy item for a message ID of WTORS. For details see the description of theOUTREP command in IBM Tivoli System Automation for z/OS Programmer'sReference.

Processing of Primary WTORsTo prevent SA z/OS from replying to primary WTORs as soon as they arereceived, the replies are not defined directly for the message ID of the primaryWTOR. Instead, the issuing of replies to primary WTORs is invoked by othermessages or executed commands.

Thus the replies for primary WTORs that are to be deferred are defined for the IDof these invoking messages, or the replies to be issued are provided for apredetermined message ID. For example, the SHUTDOWN automation policy itemis used to define the replies to be issued during shutdown.

The reply ID of any stored, primary WTOR to a subsystem can be used foroperator communication or the shutdown of this subsystem.

If SA z/OS has to communicate with a subsystem by issuing a reply but anoutstanding WTOR has not yet been stored for the subsystem, the RETRY option isused to wait for the required WTOR.

You can define multiple replies with the same pass or selection option for amessage ID. These replies can be used in response to a sequence of incomingprimary WTORs.

ExampleMessage ABC123D is issued by application ABCAPPL during startup aspermanent, outstanding WTOR and SA z/OS stores it as primary WTOR for thisapplication. During the lifetime of the application, whenever message ABC789I isissued in special situations, a reply should be issued to the permanent, outstandingWTOR ABC123D for this application. The MESSAGES/USER DATA automationpolicy item for message ID ABC789I of the application is used to define this reply.

When message ABC789I is issued by the application, SA z/OS retrieves the replyID of the permanent, outstanding WTOR and issues command MVS R 117,ABCRESTORE, as shown in Figure 31 on page 168.

WTOR Processing

Chapter 16. WTOR Processing 167

RestrictionsThe reply IDs of a subsystem's outstanding primary WTORs are stored bySA z/OS as a blank-separated list without leading zeros. The storage for this isrestricted to 255 bytes. If this limit is reached, the reply IDs of further incomingprimary WTORs are ignored.

Usage NotesWhen storing incoming WTORs, a search for code definitions for the message ID,WTORS, is first made in the entry for the subsystem that issued the WTOR.

If the subsystem itself cannot be found in the automation policy or the codedefinitions that are searched for are not found for the subsystem, they are searchedfor under the MVSESA entry. For subsystems such as IMS or NetView that have apermanent outstanding reply, you should specify the code definitions for thesubsystem entries themselves instead of MVSESA. This improves performance byreducing searches within the automation policy.

Reply defined for message ABC789I:

Entry Name : ABCAPPL Message ID :

Pass/ Retry Reply TextSelection Count__________ __ ____________________________

ABC789I

ABC RESTORE

Command issued:

MVS R ,117 ABC RESTORE

Primary WTOR stored for application ABCAPPL:

117 ABC123D REPLY WITH VALID COMMAND

Incoming message ABC789I

Figure 31. Example Processing of a Primary WTOR

WTOR Processing


Chapter 17. SA z/OS User Exits

To allow user-specific activities that are not covered by the customization dialogs,SA z/OS provides support for the following classes of user exits:v Initialization exits that are called at the start of SA z/OS initialization, before

message AOF603D is issued, see “Initialization Exits”v Static exits that are called at fixed points during SA z/OS processing, see “Static

Exits” on page 173v Flag exits that are called when SA z/OS needs to evaluate an automation flag,

see “Flag Exits” on page 175v Customization Dialog exits that can be called during certain phases when

working with the customization dialog, see “Customization Dialog Exits” onpage 178

v Command exits that can be called during the processing of certain commands,see “Command Exits” on page 182

For copying and updating sample exits refer to IBM Tivoli System Automation forz/OS Planning and Installation.

Additionally, SA z/OS has a number of facilities that behave in an exit-likemanner, see “Pseudo-Exits” on page 187.

Initialization ExitsThese exits are invoked during SA z/OS initialization:v AOFEXDEFv AOFEXI01v AOFEXI02v AOFEXI03v AOFEXI04v AOFEXI05v AOFEXI06v AOFEXINTv Environmental Setup Exits

Figure 32 on page 170 shows the sequence that exits may be invoked in duringSA z/OS initialization.


||

AOFEXDEFThis exit is called at the start of SA z/OS initialization, before message AOF603Dis issued. For example, using AOFEXDEF you can load a different MPF table.

This exit is run on AUTO1.

Parameters: None.

Return Codes: 0 is expected.

NetView initializedCNMCSSIR started

ACF COLD

MVSESA.RELOAD.CONFIRM

MVSESA.RELOAD.ACTION

AOFEXDEF initializationexit invoked

AOF603D WTOR...initialization process...

AOFEXI01 initialization exit

AOFEXINT exitinvoked or scheduled

initialization

Initialized

AOFEXI02 exitinitialization

AOF110I message issued



Environmental setupexits invoked

AOFEXSTA static exit maybe invoked from here

Flag exitsmay be invoked

Exit Sequence during SA z/OS Initialization

Figure 32. SA z/OS Exit Sequence during SA z/OS Initialization

Initialization Exits


AOFEXI01This exit is invoked before the AOF603D ENTER AUTOMATION OPTIONS replyis issued. It is invoked in a NetView PIPE and gets the data that is displayed in theAOF767I message as input in the default SAFE. With this exit you can add orremove lines from the message and add additional options to the reply.

Parameters: The Load Type is passed on input.

IPL|RECYCLEIPL Indicates that SA z/OS has been started after an IPL.RECYCLE

Indicates that NetView and therefore SA z/OS has been restarted.


AOFEXI02This exit is invoked after the operator has replied to the AOF603D reply. It gets theoperator's response to the reply as input in the default safe and it can remove, add,or change the options that the operator has entered.

Parameters: The Load Type is passed on input. See “AOFEXI01.”


AOFEXI03This exit is invoked before SA z/OS loads the NetView automation table(s) andmessage revision table. It can be used to create statistics of the currently loadedATs. Together with the AT listings that SA z/OS produces at load, these statisticscan be used for any purpose.

Parameters: None.


AOFEXI04This exit is invoked after SA z/OS loads the NetView automation table(s) andmessage revision table. It can be used to store the AT listings that SA z/OSproduces at load.

Parameters: None.


AOFEXI05This exit is called before either an ACF load or refresh takes place. The parameterindicates what action the automation agent is going to process: REFRESH orLOAD.

Parameters: Type of ACF action (REFRESH or LOAD).

Return Codes: 0 (Note: the return code is ignored by the caller).


Chapter 17. SA z/OS User Exits 171

AOFEXI06This exit is called after an ACF process (LOAD or REFRESH) has completed(AOFCOMPL=YES) and before the AOF540I message is issued.

Parameters: Type of ACF action (REFRESH or LOAD).

Return Codes: 0 (Note: the return code is ignored by the caller).

AOFEXINTThis exit is called when SA z/OS initialization is complete, before messageAOF540I is issued. You can use AOFEXINT to call your own initializationprocessing after SA z/OS has finished. Refer also to the description of the globalvariable AOFSERXINT in AOFSERXINT global variable.

Parameters: The input parameter is the Starttype which is one of the following:RESYNC, IPL, REFRESH, RELOAD, RECYCLE.


Environmental Setup ExitsThe SA z/OS customization dialog allows you to define a string of exits that areinvoked during SA z/OS initialization processing.

These exits are defined using the SYSTEM INFO policy item of the System (SYS)entry type. See IBM Tivoli System Automation for z/OS Defining Automation Policy formore information.

Environmental setup exits are invoked after SA z/OS has started its various tasks,but before the primary automation table has been loaded. You can use these exitsto initiate your own automation, but some SA z/OS services may be unavailablebecause SA z/OS has not yet finished initializing when these exits are called. Inparticular, status information may be inaccurate because SA z/OS may not havefinished resynchronization. Environmental setup exits run on AUTO1.

Parameters

Parameters are passed in sequence, delimited by blanks.

INITIALIZATIONINITIALIZATION is a constant.

RELOAD|REFRESH|IPL|RECYCLERELOAD

Indicates that the automation control file has been reloaded.REFRESH

Indicates that the automation control file has been refreshed.IPL Indicates that SA z/OS has just been restarted after a system IPL.RECYCLE

Indicates that NetView has been restarted.

Return Codes

0 is expected. If you return a non-zero return code you may prevent other exitsfrom being invoked or disrupt SA z/OS initialization.



Usage Notesv These exits are not driven if you run RESYNC.v Unlike the other static exits, you must specify the name of the routine or

routines to invoke in the automation control file.

Static ExitsThese exits are invoked at fixed points in SA z/OS processing.

They are always invoked if they are found in the DSICLD concatenation. Positivereturn codes from these exits are generally ignored, though it is recommended thatyou always exit with a return code of 0.

The main purpose of static exits is to allow you to take your own actions atspecific points during SA z/OS processing. The static exits available are describedbelow.

AOFEXSTAThis exit is called from AOCUPDT every time the automation status of anapplication is updated.

Note: It is not necessary for AOCUPDT to change an application automation statusfor this exit to be called. The exit is still invoked if the update does not result in achange of status.

AOFEXSTA can be used to perform any special status transition processing thatcannot be triggered by other methods.

Note: This exit is invoked frequently, and is invoked at times when SA z/OS isnot fully initialized. Your exit code should be as robust and efficient as possible.

SA z/OS attempts to load AOFEXSTA into storage at initialization. If this attemptfails, AOFEXSTA is not invoked on any AOCUPDT calls. To activate the exit itmust be present in the DSICLD concatenation when the automation control file isloaded or reloaded.

AOFEXSTA runs on the task that called AOCUPDT, after all other processing hasfinished.

Attention: AOFEXSTA is scheduled with EXCMD opid(). If your operators areissuing commands that change application statuses and you want to useAOFEXSTA, you may have to modify your command authorization definitions.

Parameters: Parameters are passed in sequence, delimited by commas.

Resource typeSA z/OS uses types of SUBSYSTEM, MVSESA, WTORS, and SPOOL. Otherusers may use other resource types.

Resource NameFor an application, this is the name of the subsystem it is defined as.

Automation StatusFor an application, this is one of the automation statuses that is supported bySA z/OS.



SDF RootThis is the SDF Root, as specified in the customization dialog, for the systemthat originated the status update. Generally the exit is driven only for statuschanges on other systems on the automation focal point.


Restrictions:

v Because the exit is scheduled with EXCMD, the status update and subsequentprocessing in the caller will have completed before the exit is invoked.

v Check the resource type and the SDF root to ensure you are only trying toprocess the right things.

v Plan carefully before you take any action to change the status of an applicationfrom this exit. If you are not careful you may create a loop (AOCUPDT toAOFEXSTA to AOCUPDT to AOFEXSTA).

Notes:

1. Consider using ISSUEACT or status change commands as alternatives toAOFEXSTA, because AOFEXSTA is invoked for every status update thatseriously degrades performance.

2. If the advanced automation options are set up appropriately, the ACTIVMSGand TERMMSG commands issues commands whenever an application changesto a particular status. It may be more appropriate to place commands here,rather than in the status change exit, which gets driven for every status updateof every resource. It is recommended to use status change commands for betterperformance.

AOFEXX02The exit allows the installation to decide whether or not an SDF update should beperformed for the specified resource.

A non-zero return code from the exit causes the SDF update processing to beskipped, both locally as well as for the focal point.

This exit is called prior to posting entries to SDF to provide the facility to filter outspecific events.

Refer to the sample exit for details of the parameters passed to the exit and thereturn codes.

AOFEXX03The exit allows the installation to decide whether or not status change notificationshould be forwarded to the NMC focal point for the specified resource.

A non-zero return code from the exit causes status change forwarding to beskipped.

This exit is called prior to posting entries to NMC to provide the facility to filterout specific events.


Static Exits


AOFEXX04This exit is called from CHKTHRES every time that this routine is called to checkthe number of errors recorded in the automation status file for a given resourceagainst error thresholds that are defined in the automation control file.


AOFEXX05The exit is driven by SA z/OS at initialization time when setting up the SDFpanels and trees or by the RESYNC SDFDEFS command.

The exit is used by the installation to replace user variables in the SDF paneldefinition. A user variable must follow the same convention as a z/OS systemsymbol, that means it must start with an ampersand (&) and finish with a dot(.).An example is &MVDOMAIN.


AOFEXX15This exit allows you to write a log entry for each status change notification thatarrives at the NMC focal point.

Refer to the sample exit for details of the parameters passed to the exit.

Flag ExitsUsing automation flag exits you can cause your automated operations code to exitnormal SA z/OS processing to an external source, such as a scheduling function,to determine whether automation should be on or off for a given resource at thatparticular instant.

Flag exits can be defined for any flag (AUTOMATION, INITSTART, START,RECOVERY, TERMINATE, or RESTART) on any major or minor resource. See thedescription of the MINOR RESOURCES policy item in IBM Tivoli SystemAutomation for z/OS Defining Automation Policy for more information on minorresources.

You can specify multiple exits for each flag.

A flag exit is invoked only if SA z/OS checks the value of the current flag settingduring the flag evaluation process of AOCQRY, as described in IBM Tivoli SystemAutomation for z/OS Programmer's Reference. If one of the global or specific flags,which have to be checked in one iteration step during the evaluation process overthe inheritance hierarchy levels is set to NO, the other flag no longer has to bechecked.

With the default BYPASS option of AOCQRY, exits that have been defined for theautomation flag of a resource are executed when that automation flag is checkedduring flag evaluation and the flag value is EXITS.

Static Exits


With the FORCED option of AOCQRY, exits that have been defined for theautomation flag of a resource are executed when that automation flag is checkedduring flag evaluation, independent of the flag value, as long as it is not empty.

If an automation flag is set to EXITS, the flag value is assumed to be YES duringflag checking as long as none of the exits that have been defined for the checkedresource switch the flag to NO. Exits that are forced to execute do not change theflag value.

Flag settings are determined by:v The automation policy settingsv NOAUTO periods (the flag is OFF during a NOAUTO period)v The user-entered INGAUTO command

For example, if you enter the following flag settings:

Resource Flag Setting

DEFAULTS AUTOMATION YES

SUBSYSTEM RESTART NO

JES2 AUTOMATION Exit J2AUT

JES2 START Exit J2STR

JES2 RECOVERY NO

JES2 TERMINATE Exits J2SD1 and J2SD2

The effective flags for JES2 are:

Flag Effective setting

AUTOMATION YES, Exit J2AUT

INITSTART YES

START YES, Exit J2STR

RECOVERY NO

TERMINATE YES, Exits J2SD1 and J2SD2

RESTART NO

When SA z/OS checks the current value of any flag for the JES2 application, theprocess is as follows:

Flag Process

AUTOMATION 1. Call exit J2AUT.

2. If the exit returns:

v RC=0: AUTOMATION flag is YES

v RC>0: AUTOMATION flag is NO

INITSTART 1. Call exit J2AUT (because of AUTOMATION global flag).

2. If the exit returns:

v RC=0: INITSTART flag is YES

v RC>0: INITSTART flag is NO

Flag Exits


Flag Process

START 1. Call exit J2AUT (because of AUTOMATION global flag).

2. If exit returns RC=0, call exit J2STR.

3. If:

v Both flags return RC=0: START flag is YES

v Either flag returns RC>0: START flag is NO

RECOVERY RECOVERY flag is NO

TERMINATE 1. Call exit J2AUT (because of AUTOMATION global flag).

2. If exit returns RC=0, call exit J2SD1.

3. If exit J2SD1 returns RC=0, call exit J2SD2.

4. If:

v Both flags return RC=0: TERMINATE flag is YES

v Either flag returns RC>0: TERMINATE flag is NO

RESTART 1. Call exit J2AUT (because of AUTOMATION global flag).

2. Regardless of the return code, the RESTART flag is NO.

Note: Normally the START and RECOVERY flags are checked by SA z/OS onlyfor minor resources but not for the subsystem itself.

ParametersParameters are supplied in sequence, delimited by blanks.

FlagThis is the name of the flag that is being checked. Possible values areAUTOMATION, INITSTART, START, RECOVERY, TERMINATE or RESTART.

Time SettingTime Setting is a constant. It can be either:

AUTOAutomation is currently turned on.

NOAUTOAutomation is currently turned off.

A value of NOAUTO is possible only if AOCQRY is called with the parameterEXITS=FORCED.

Note: This ensures that the exit is invoked, but it is not possible for an exit tooverride a NOAUTO period.

Resource NameThis is the name of the resource that the flag is being requested for. For minorresources it contains the fully qualified minor resource name. Given no flagdefinition for TSO.USER.MAG1 and an exit enabled for TSO.USER, theresource name passed to the exit would be TSO.USER.MAG1 if a check wasmade for TSO.USER.MAG1.

Resource TypeThis is the type of the resource that the flag is being requested for. Possiblevalues are DEFAULTS, SUBSYSTEM, or MVSESA (the value of the commonglobal variable AOFSYSTEM).

Flag Exits


Target PrefixThis is the TGPFX value with which AOCQRY was invoked. If TGPFX is notspecified, the value SUB is passed.

Task Global VariablesThe task global variables that are set by the AOCQRY command are available inflag exits.

Return Codes0 Automation is allowed by the exit.

> 0 Automation is not allowed.

Notes:

1. Flag exits are always called through the AOCQRY command. This means thatthe task global variables for the resource have been primed and are availablefor use. Normally the names of the task global variables are prefixed with SUB,but if AOCQRY is called with a different value for parameter TGPFX, they arefound in variables that are prefixed with that value. You should use the TGPFXparameter that is passed to locate the task global variables.

2. The set of task global variables that are set by AOCQRY depends on the valuesfor the resource and request parameter. Make sure that the task global variablesthat you rely on in your exit are being set up.

3. If an exit is invoked for a minor resource, the task global variables are set forthe major resource that is associated with that minor resource.

4. If you call AOCQRY from within your exit you must specify a TGPFX valuethat is different from the TGPFX parameter value you were passed. You areresponsible for ensuring the uniqueness of all TGPFXs if you nest AOCQRYexits. Because this can become quite complex, it is recommended you avoidnesting exits.

5. Do not code calls to ACFCMD, ACFREP, or CDEMATCH because these usetask global variables that are prefixed with SUB that may not be set up for theapplication that you want to process.

6. Do not change any of the task global variables that have been set by AOCQRY.7. Flag exits may be called frequently, so performance is important.8. If AOCQRY is specified with FORCE and multiple exits are defined for a flag,

the exits are called in order.

Customization Dialog ExitsSA z/OS provides a series of the following user exits that can be invoked duringcertain phases while working with the customization dialog.v “User Exits for BUILD Processing” on page 179v “User Exits for COPY Processing” on page 180v “User Exits for DELETE Processing” on page 180v “User Exits for CONVERT Processing” on page 181v “User Exits for RENAME and IMPORT Functions” on page 181

“Invocation of Customization Dialog Exits” on page 182 provides information onhow to activate the user exits.

Flag Exits


User Exits for BUILD ProcessingThe following user exits are provided for the process of building the automationcontrol file.v INGEX10: This is called before the automation control file build function starts.

This exit is only available when the build process is initiated from thecustomization dialogs.

v INGEX01: This is called before the automation control file build function starts.This exit is available when the build process is initiated from the customizationdialogs, from a batch job submitted via the customization dialogs, or from abatch job submitted independently from the customization dialogs.When a BUILD mode of BATCH is selected in the customization dialogs, theJCL for the batch job is submitted and INGEX01 is called when the job beginsexecution and before the automation control file build function starts in batch.

v INGEX02: This is called after the configuration file build has ended. This exit isavailable when the BUILD process is initiated from the customization dialogs,from a batch job submitted through the customization dialogs, or from a batchjob submitted independently from the customization dialogs.

The following parameters are passed to both INGEX01 and INGEX02 exits,separated by commas:v Parm1 = PolicyDB namev Parm2 = Enterprise namev Parm3 = BUILD output data setv Parm4 = entry type (or blank)v Parm5 = entry name (or blank)v Parm6 = BUILD type (MOD/ALL)v Parm7 = BUILD mode (ONLINE/BATCH)v Parm8 = Configuration (0=NORMAL/1=ALTERNATE)v Parm9 = Sysplex name (or blank)v Parm10 = Build option (1,2, or 3)v Parm11 = return code (for INGEX02 only)

If user exit INGEX10 produces return code RC = 0, build processing continues. If areturn code RC > 0 is produced, an error message is returned and the buildprocessing terminates.

If user exit INGEX10 ends with return code RC > 0, user exits INGEX01 andINGEX02 are not called. Processing terminates.

If user exit INGEX10 ends with return code RC > 0 and a BUILD mode of BATCHwas selected in the customization dialogs, no JCL is submitted to run the build inbatch. Processing terminates.

If user exit INGEX01 produces return code RC = 0, build processing continues. If areturn code RC > 0 is produced, an error message is returned and build processingterminates. If the build is run in batch mode, and a return code RC > 0 isproduced, the job finishes with a return code RC 08.

If user exit INGEX01 ended with return code RC > 0, user exit INGEX02 is notcalled because the build function was not started. Processing terminates.

User exit INGEX02 is always called when the BUILD process has started,irrespective of whether it has completed or not.

Customization Dialog Exits


If user exit INGEX02 produces a return code RC > 0, an error message isdisplayed. If the build is run in batch mode, and a return code RC > 0 is produced,the job completes with a return code RC 04. If a severe build error occurred, thejob completes with a return code RC 20.

The return codes and their meaning are as follows:

0 Successful

4 Build with minor errors

12 No build (data is inconsistent)

20 No build (severe errors)

User Exits for COPY ProcessingTwo user exits are implemented for the COPY processing.1. INGEX03: This is called before the COPY function starts. The following

parameters are passed:v Entry name of the entry to be copied to (target)v Entry name of the entry to be copied from (source)v Entry type (for example, APL)

2. INGEX04: This is called after the COPY function has ended. The followingparameters are passed:v Entry name of the entry to be copied to (target)v Entry name of the entry to be copied from (source)v Entry type (for example, APL)v Indicator whether the COPY process was successful or not (S=successful,

U=unsuccessful)

If user exit INGEX03 produces return code RC = 0, COPY processing continues. Ifa return code RC > 0 is produced, an error message is displayed, the COPYfunction does not start, and processing terminates.

If user exit INGEX03 ended with return code RC > 0, the user exit INGEX04 is notcalled because the COPY processing will terminate.

User exit INGEX04 is always called once the COPY function has started. Theinformation about the success or failure of the COPY function is passed as aparameter.

If user exit INGEX04 produces a return code RC > 0, an error message isdisplayed.

User Exits for DELETE ProcessingTwo user exits are implemented for the DELETE processing.1. INGEX05: This is called before the DELETE process starts. The following

parameters are passed:v Entry name of the entry to be deletedv Entry type (for example, APL)

2. INGEX06: This is called after the DELETE process has ended. The followingparameters are passed:v Entry name of the entry to be deleted



v Entry type (for example, APL)v Indicator whether the DELETE process was successful or not (S=successful,

U=unsuccessful)

If user exit INGEX05 produces return code RC = 0, the DELETE processingcontinues. If a return code RC > 0 is produced, an error message is displayed, theDELETE function does not start and the processing terminates.

If user exit INGEX05 ended with a return code RC > 0, user exit INGEX06 is notcalled because the DELETE processing will terminate.

User exit INGEX06 is always called once the DELETE function has started. Theinformation about the success or failure of the DELETE function is passed as aparameter.


User Exits for CONVERT ProcessingTwo user exits are implemented for the CONVERT processing.1. INGEX07: This is called before the CONVERT process starts. No parameters are

passed.2. INGEX08: This is called after the CONVERT process has ended. No parameters

are passed.

If user exit INGEX07 produces return code RC = 0, the CONVERT processingcontinues. If a return code RC > 0 is produced, an error message is displayed, theCONVERT function does not start and the processing terminates.

If user exit INGEX07 ended with a return code RC > 0, user exit INGEX08 is notcalled because the CONVERT processing will terminate.

User exit INGEX08 is always called once the CONVERT function has started.


User Exits for RENAME and IMPORT FunctionsThe following user exits are provided for the renaming and import functions.v INGEX15: This is called before an entry is renamed. The following parameters

are passed:– Entry Name– Entry Type

Note: When renaming an entry, a new entry is created and the old entry isdeleted. Therefore, INGEX05 is called before the old entry is deleted andINGEX06 is called after the entry has been deleted.

v INGEX16: This is called after an entry has been renamed. The followingparameters are passed:– Entry Type– Old Entry Name– New Entry Name



|||

v INGEX17: This is called during the IMPORT function, when reading data fromthe source policy database. One parameter is passed:– Name of copy data work table. This table contains the entry types and entry

names of the data to be copied.v INGEX18: This is called after the IMPORT function has ended. INGEX18 is only

called if INGEX17 was called at the beginning of the IMPORT function. If checkshave been made that prevent INGEX17 being called, INGEX18 is not calledeither.One parameter is passed:– Indicator whether the IMPORT process was successful (S=successful,

U=unsuccessful)

User Exits for Other FunctionsThe following user exits are provided for other functions.v INGEX09: This is called when the log data set is switched, usually because the

current data set is full. One parameter is passed:– Name of current log data set, for example, the data set that went out of space

v INGEX20: This is called after the links have been changed. No parameters arepassed.

v INGEX21: This is called before the policy database report is invoked. Noparameters are passed.

Invocation of Customization Dialog ExitsThe user exits are part of the SA z/OS product. Therefore they are supplied in thesame data set as all other ISPF REXX modules (part of SINGIREX). All of thesupplied samples just perform a 'RETURN' with return code RC=0.

You have two possibilities to apply your user modifications:1. Edit the user exit (or exits) in the supplied library. Your changes do not have

any consequences for the code of the SA z/OS product. These exits are notserviced (via PTF) by IBM because they do not include any code at the time ofproduct delivery.

2. Supply the modified user exit in a private data set. Then you have toconcatenate your private data set to your SYSEXEC library chain. As INGDLGsupports multiple data set names specified for ddname SYSEXEC, this can bedone in the following way:INGDLG SELECT(ADMIN) ALLOCATE(YES) HLQ(SYS1)

SYSEXEC(usr.private.dsn SYS1.SINGIREX)

This example assumes that the high level qualifier of the data sets where theIBM supplied parts exist is SYS1.If you specify the SYSEXEC parameter in the INGDLG call, you need to specifythe IBM supplied library explicitly with its fully qualified data set name.

Command ExitsThese exits can be called during the processing of certain commands.

Sample exits are provided with the product. For details on copying and updatingsample exits, refer to IBM Tivoli System Automation for z/OS Planning andInstallation.



|||

AOFEXC00The AOFEXC00 exit routine is called if the selection L has been entered in the AOCcommand dialog. No parameters are passed to the routine. The purpose of thisroutine is to act as the starting point for installation-provided local functions.

AOFEXC01If this exit is defined, it is invoked during INGREQ processing before Precheck andVerification processing.

The exit allows you to modify the parameters that are passed.

Refer to the sample exit for details of the parameters that are passed to the exitand the return codes.

The exit is not called during the INGREQ REQ=CANCEL processing. You may useexit AOFEXC08 instead.

AOFEXC02If this exit routine is defined, it is invoked during INGSCHED processing beforethe schedule override file is updated. The parameters are positional and separatedby a comma.


AOFEXC03If this exit routine is defined, it is invoked by the DISPINFO command slave toretrieve user-supplied information about the subsystem. The input for the routineis the subsystem name. The data returned by the exit is shown as part of theDISPINFO output.


AOFEXC04If this exit routine is defined, the command code U is supported for the DISPMTR,DISPSTAT, and INGLIST commands. The input for the AOFEXC04 exit is theresource name (subsystem name for DISPSTAT) and the location of the resource.The location is either the system name if the resource resides on a system memberof the local sysplex, or the domain ID if the resource resides on a system which isoutside of the local sysplex. The parameters are separated by a comma.


AOFEXC05This exit is called on entry of the INGLIST command. The exit allows you tomodify the input parameters. The modified input parameters are returned to theINGLIST command by sending a message (single or multiline) to the console, forexample:OBSERVED=* DESIRED=*

Command Exits


||


AOFEXC06This exit is called on entry of the INGSET command with the SET action. The exitallows you to perform authorization checking of the resources for the INGSETcommand.


AOFEXC07This exit is called on entry of the INGIMS command. The exit allows you toperform authorization checking of the IMS subsystem that is the subject of theINGIMS command.


AOFEXC08This exit is called on entry of the INGVOTE command. The exit allows you toperform authorization checking of the resources for the INGVOTE command.Because the INGSET CANCEL/KILL action uses the INGVOTE command, this exitis also called when performing this action.


AOFEXC09This exit is called on entry of the SETSTATE command. The exit allows you toperform authorization checking of the resources for the SETSTATE command.


AOFEXC10This exit is called on entry of the INGEVENT command. The exit allows you toperform authorization checking of the resources for the INGEVENT command.


AOFEXC11This exit is called on entry of the INGCICS command. The exit allows you toperform authorization checking of the resources for the INGCICS command.


Command Exits


AOFEXC13This exit is called on entry to the INGGROUP and INGMOVE commands. The exitallows you to perform authorization checking of the user ID that issues thecommand.


AOFEXC14This exit is called by the SA z/OS GDPS termination routine (INGRGDPS) afterstopping the PAM or selecting a SAM to become the PAM.

Refer to the sample exit for details of the return codes.

AOFEXC15If this exit routine is defined, it is invoked during INGREQ processing after theGO confirmation has been received.

The user exit is called in a PIPE. Refer to the sample exit for details of theparameters that are passed to the exit.

AOFEXC16This exit is invoked by the INGTHRES command prior to updating or deleting thethresholds for a given resource. It allows you to perform authorization checking ofthe requested action. If the exit returns with a non-zero return code and additionaldata is written to the console, this data is shown in the message panel. If noadditional data is passed back in the exit, message AOF227I is issued.


AOFEXC17This exit is invoked by the INGALERT command. It allows you to:v Modify the event textv Reduce the Inform List with event notification targets such as IOM, EIF, TTT,

and USRv Modify the value that is returned from the matching code definition with

information such as the event severity or whether to ignore the event


AOFEXC18This exit is invoked by the INGLKUP command. It is driven prior to stopping orcanceling the specified address space. It allows you to perform authorizationchecking of the requested action. If the exit returns with a non-zero return codeand additional data is written to the console, this data is shown in the messagepanel. If no additional data is passed back in the exit, message AOF227I is issued.


Command Exits


AOFEXC19This exit is invoked by the INGAMS command. It is driven in the following cases:v Enabling or disabling access to the takeover filev Suspending or resuming systemsv Refreshing the configurationv Performing a diagnostic action (starting or stopping recording, taking a

snapshot)v Switching the primary automation manager

The exit allows you to perform authorization checking of the INGAMS command.If the exit returns with a non-zero return code and additional data is written to theconsole, this data is shown in the message panel. If no additional data is passedback in the exit, message AOF227I is issued.


AOFEXC20This exit is called when a command is passed via the TWS request interface. Theexit allows the installation to perform authorization checking. Optionally, the exitcan modify the command and/or the completion information by returning up totwo data lines:v line 1 is the modified command including all its parameters. A null string must

be returned when the command is not modified. This is only necessary whenmodifying the completion information via the exit

v line 2 is the completion information:– maximum return code– completion checking coding.

The parameters must be separated by a comma. Error code U010 will be postedwhen one of the parameters is wrong.

The installation exit is called in a PIPE. If the exit returns a bad return code andadditional data is written to the console, this data is written in the netlog. If noadditional data is passed in the exit, message AOF227I is issued.


AOFEXC21This exit is invoked by the INGOPC command with the option REQ=MOD. Itallows you to perform authorization checking of the requested action. If the exitreturns with a non-zero return code and additional data is written to the console,this data is shown in the message panel. If no additional data is passed back in theexit, message AOF227I is issued.


Command Exits


AOFEXC22This exit is called when a trouble ticket is created using the INGALERT command.It allows you to determine the trouble ticket detail data that is to be written to thedetail data set.


AOFEXC23This exit is invoked when a request is passed via the TWS interface. It allows youto perform authorization checking of the requested action. If the exit returns anon-zero return code and additional data is written to the console, this data istaken as a message. If no additional data is passed back in the exit, messageAOF227I is issued.


AOFEXC24If this exit is defined, it is invoked during INGRUN processing. This exit allowsyou to modify the parameters that are passed.


AOFEXC25

This exit is invoked when an INGAMS REFRESH request is processed. It providesdetails about new, changed and deleted subsystems (APLs), application groups(APGs) and monitor resources (MTRs). The exit is driven on NetView operatorAUTO2 to prevent performance degradation during refresh processing. It isrecommended to pass back a return code of 0, however, the return code is notchecked, today.

Refer to the sample exit for further details.

Pseudo-ExitsThis section discusses a number of places where SA z/OS either makes special useof a flag exit or has a function with certain, exit-like, qualities.

Automation Control File Reload Permission ExitWhen an operator issues the ACF COLD command, SA z/OS checks the globalAUTOMATION flag of minor resource MVSESA.RELOAD.CONFIRM.

If the flag is set to NO, the automation control file reload is not allowed. If the flagis set to YES, the task global AOFCONFIRM is checked. If AOFCONFIRM has beenset to a non-null value, the user is prompted to confirm that they want theautomation control file to be reloaded.

Notes:

1. Note that an exit can be associated with the global AUTOMATION flag for thisresource.

Command Exits


|

||||||

|

2. An automation control file cannot be loaded if the global AUTOMATION flagfor the major resource MVSESA is set to N. If the global AUTOMATION flagfor the minor resource MVSESA.RELOAD.CONFIRM is set to Y, reloading theACF is permitted.

Automation Control File Reload Action ExitAfter the automation control file reload permission exit is checked, when SA z/OSis committed to reloading the automation control file, it checks the globalAUTOMATION flag for minor resource MVSESA.RELOAD.ACTION.

The actual setting of this flag (ON or OFF) is ignored, but any exits defined for itare invoked. All exits should return a return code of 0.

Subsystem Up at Initialization CommandsYou can specify the commands to be issued if SA z/OS finishes resynchronizingstatuses and an application is found to be up. These commands are specified in theCustomization Dialog > APL entry type > STARTUP policy > REFRESHSTARTphase.

These commands can be useful for synchronizing local automation that has beenbuilt on top of SA z/OS.

Testing ExitsExits should be well tested with a variety of different input parameters before theyare put into production. For exits that need AOCQRY task globals, you can callAOCQRY to set up the globals without evaluating the flag exits, and then invokethe exit on its own for testing purposes. This method saves the overhead of callingAOCQRY every time you run the exit.

Attention:

If you have a syntax error or a no-value-condition in your exit it can cause parts ofSA z/OS to abend, resulting in severe disruption of your automation.

Pseudo-Exits


Chapter 18. Automation Solutions

SA z/OS provides solutions that enable automatic processing of z/OScomponents, data sets and job scheduling systems as well as automationprocedures that are useful tools in the automation processing context. By usingthese prefabricated automation procedures you can save the time to develop yourown procedures to handle the processing in corresponding situations.

In particular these automation routines provide solutions for:v “LOGREC Data Set Processing” on page 190v “SMF Data Set Processing” on page 190v “SYSLOG Processing” on page 190v “System Log Failure Recovery” on page 191v “SVC Dump Processing” on page 191v “Deletion of Processed WTORs from the Display” on page 191v “AMRF Buffer Shortage Processing” on page 191v “JES2 Spool Monitoring” on page 65v “Drain Processing Prior to JES2 Shutdown” on page 192v “IMS Transaction Recovery” on page 192v “Looping Address Space Suppression” on page 192

The solutions for automatic processing of these situations include definitions in theautomation configuration files and automation procedures.

It is common to all the solutions that are provided that the automation proceduresfirst determine whether automation is allowed by checking the correspondingautomation flags with the AOCQRY command. See IBM Tivoli System Automationfor z/OS Defining Automation Policy for further information concerning types andsettings of automation flags. Use the DISPFLGS command to display ortemporarily change the current settings of the automation flags.

Some of the automation routines respond to messages by issuing commands fromthe automation configuration files. Most of these automation routines keep track ofthe reception of these messages and compare the frequency of the incomingmessages with predefined thresholds of infrequent, frequent, and critical levels. Ifsuch a defined threshold is exceeded, it is used as the option for selecting theappropriate commands according to the first field in the command entry of theMESSAGES/USER DATA policy item of the configuration file. If no threshold isexceeded the commands defined for the selection option ALWAYS are issued. See“How SA z/OS Uses Error Thresholds” in IBM Tivoli System Automation for z/OSUser's Guide for further information on setting up thresholds.

This chapter describes the details of the automation functions that are providedwith SA z/OS.


|

LOGREC Data Set ProcessingThe logrec recovery function responds to system messages that indicate that thelogrec data set is full or nearly full. The recovery function issues predefinedcommands to dump and clear the logrec data sets. While the recovery function isin progress, it prevents the automation processing being started a second time.

The logrec recovery function includes the following items:v Automation routines AOFRSA01 and AOFRSA02, see “AOFRSA01” on page 198

and “AOFRSA02” on page 199v Automation table entries for system messages IFB040I, IFB060E, IFB080E,

IFB081I, and IFC001Iv Error threshold definitions for MVS component minor resource LOGRECv Command specification in the MESSAGES/USER DATA automation policy item

for the special message ID LOGREC

SMF Data Set ProcessingThe SMF recovery function that is provided responds to system messages thatindicate that the SMF data set is full or has been switched. Predefined commandsfrom the configuration files are selected to dump and clear the contents of the SMFdata set.

The commands to be selected can be defined depending on the occurrence of theincoming messages. Additionally SA z/OS checks at initialization time whetherSMF data sets need to be dumped and triggers the appropriate recovery action.The SMF recovery function includes the following items:v Automation routine AOFRSA03, see “AOFRSA03” on page 201v Automation table entries for system messages IEE362A, IEE362I, IEE391A and

IEE392Iv Error threshold definitions for MVS component minor resource SMFDUMPv Command specification in the MESSAGES/USER DATA automation policy item

for the special message ID SMFDUMP

SYSLOG ProcessingThe syslog function that is provided responds to messages that are queued to thesyslog. The function starts an external writer to save the syslog that was queued.The commands to be selected can be defined depending on the occurrence of theincoming messages.

The syslog function includes the following items:v Automation routine AOFRSA08, see “AOFRSA08” on page 203v Automation table entry for system message IEE043Iv Error threshold definitions for MVS component minor resource SYSLOGv Command specification in the MESSAGES/USER DATA automation policy item

for the special message ID SYSLOG

LOGREC Data Set Processing


System Log Failure RecoveryThe system log failure recovery function that is provided responds to a system loginactive message by restarting the system log. If the system log should be availableto be used as the hardcopy medium, the recovery function assigns the system logas the hardcopy medium.

The recovery commands are only issued if the occurrence of the system loginactive message that is received does not exceed a defined critical threshold.

The system log failure recovery function that is provided includes the followingitems:v Automation routine INGRX740, see “INGRX740” on page 223v Automation table entries for system messages IEE037D, IEE041I, IEE533E,

IEE769E, IEE043Iv Recovery automation flag for the MVS component minor resource LOGv Error threshold definitions for the MVS component minor resource LOGv Command specification in the MESSAGES/USERDATA automation policy item

for the special message ID LOG

SVC Dump ProcessingThe SVC dump processing function that is provided responds to an SVCdump-taken message by issuing predefined commands from the configuration filesto handle the dump. The commands to be selected can be defined depending onthe occurrence of the incoming messages.

The provided SVC dump processing function includes the following items:v Automation routine AOFRSA0C, see “AOFRSA0C” on page 205v Automation table entries for system messages IEA611I and IEA911Ev Error threshold definitions for MVS component minor resource MVSDUMPv Command specification in the MESSAGES/USER DATA automation policy item

for the following special message IDs:– MVSDUMP– MVSDUMPTAKEN– MVSDUMPRESET

Deletion of Processed WTORs from the DisplayThe WTOR processing function that is provided deletes WTORs from SA z/OSdisplay capabilities when they are replied to or canceled.

The WTOR processing function includes the following items:v Automation routine AOFRSA0E, see “AOFRSA0E” on page 208v Automation table entries for system messages IEE400I and IEE600I

AMRF Buffer Shortage ProcessingThe AMRF buffer shortage processing function that is provided responds tomessages that report buffer shortage of the action message retention facility(AMRF). The function issues commands from the configuration files to processbuffer shortage automation.

System Log Failure Recovery

Chapter 18. Automation Solutions 191

The AMRF buffer shortage processing function that is provided includes thefollowing items:v Automation routine AOFRSA0G, see “AOFRSA0G” on page 209v Automation table entries for system messages IEA359E, IEA360A and IEA361Iv Command specification in the MESSAGES/USER DATA automation policy item

for the following special message IDs:– AMRFSHORT– AMRFFULL– AMRFCLEAR

Drain Processing Prior to JES2 ShutdownSA z/OS provides functions for drain processing of JES2 resources prior to JES2shutdown.

The JES2 drain processing function that is provided includes the following items:v Automation routines AOFRSD07, AOFRSD0F, AOFRSD0G. See “AOFRSD07” on

page 211, “AOFRSD0F” on page 213 and “AOFRSD0G” on page 215.v Automation table entries for system message HASP607.v Specifications in the JES2 DRAIN automation policy item for the JES2

applications that are to be drained and how they are to be drained prior to JES2shutdown.

IMS Transaction RecoverySA z/OS provides an IMS transaction recovery function. This responds to an IMSapplication program abend message by issuing predefined replies or commandsfrom the configuration files for recovery purposes. A recovery action is not issuedif the program is excluded from recovery processing, or the occurrence of theincoming message exceeds a predefined critical threshold.

The IMS transaction recovery function that is provided by SA z/OS includes thefollowing:v A NetView automation table entry for the application program abend message,

DFS554Av The subsystem that issues the abend message has the following automation

policy definitions:– Error threshold definitions in the MINOR RESOURCE policy item for the

minor resource PROG.progid or TRAN.tran

– Code definitions in the MESSAGES/USER DATA policy item for the messagetypes ABCODEPROG.progid, ABCODEPROG, ABCODETRAN.tran, orABCODETRAN

– Reply or command specifications in the MESSAGES/USER DATA policy itemfor the message ID DFS554A

Looping Address Space SuppressionThe Looping Address Suppression is implemented as a special APL. You canimport it from the supplied sample automation policies.

AMRF Buffer Shortage Processing


|

||

Once imported you need to add a definition for your TEMS server, perform somecustomization for the APL and then link it to the system that it will run on with anAPG.

When SA z/OS loads the SOCNTL files containing the APL, it will activate theLooping Address Space Suppression procedure.

PreparationYou need OMEGAMON for z/OS installed and running on the system where youwant to run the procedure.

You need a TEMS server up and running that is in communication withOMEGAMON for z/OS.

Check you can display:v OMEGAMON XE for z/OSv OMEGAMON data from a TEP connected to the TEMS.

You need to know:v a userid and a password that gives you access to the TEMSv the IP address and the port number of the SOAP server running on the TEMS.

If you are going to use HTTPS for communication, you need to have already set itup under TCPIP. (See the step for configuring Tivoli Enterprise Portal in"Traditional SA z/OS Configuration" in IBM Tivoli System Automation for z/OSPlanning and Installation).

Copy the Sample PolicyIf you are creating a new PDB, you can simply copy the entire *ITM add-on intoyour PDB.

If you have an existing PDB that includes some or all of the elements in the *ITMadd-on, you can be more selective:1. Take a backup of your normal PDB and then open it in the customization

dialogs.2. Go into Data Management (Option 5) and select Import for Add On (Option 2).3. Place a C (for Customize) against the *ITM add-on to get to the component

selection panel.4. You should see a list of components, use M to reMove and S to Select to ensure

that only the 'Monitoring Analytic' component is selected.5. Press PF3 and then select 1 to view the data to be imported. Select option 3 to

run the import.

After some informational messages, you should see Import Successful, after whichyou press PF3 twice to access the main menu.

Customizing your SOAP Server1. Open your PDB and select NTW (Networks).2. Enter new TEMSserver, where TEMSserver is a unique name to identify the

TEMS server that is providing your OMEGAMON data.3. Use Copy to copy the definitions from the SOAP_SERVERS network definition.

IMS Transaction Recovery


|||

||

|

||

||

|

|

|

|

|

|

||||

|

||

||

||

|

||

||

||

||

|

|

||

|

4. Select the SOAP SERVER policy and edit the HUBTEMS entry.5. Enter the Host name and the Protocol. If you are using HTTPS as your protocol,

you need to change the port number to 3661 (although this and the 1920 portnumber are only the default values, your site may have changed them).

6. Specify the User ID and Password of SAFPW. This instructs System Automationthat you are using the INGPW utility under NetView to supply the password.

7. Press PF3 twice to return to your policy panel and then select WHERE USED.8. Select the systems that will be receiving their OMEGAMON data through this

TEMS server. Note that it is important that there is only one single networkwith a HUBTEMS SOAP server defined which is linked to each system.

9. Press PF3 to return to the entry selection panel.

Configuring your APL

Copying the sample

Select the APL entry type.

Enter new LOOPSUPP_identity to create a new APL. The identity could be a TEMSname or, if you have systems from multiple sysplexes feeding into the same TEMS,a sysplex name. Doing the latter means you need to specify a different set ofpolicies for each sysplex.

Give it a SUBSYSTEM name of LOOPSUPP, press Enter and then press PF3.

Use the COPY policy to copy data from the LOOPSUPP APL.

Minor Resources

There is a minor resource called MONITOR defined. This controls the execution ofthe monitor when the APL is Available. By default its RECOVERY flag is set toLOG. This means disruptive commands are written to the Netlog rather than beingexecuted. Non-disruptive recovery actions WARN and DIAG are still performed.

For your first deployment you probably want to keep this value set to LOG, butfor future deployments you may want to enable your actions and set it to YES.

MESSAGES/USER DATA

There are two categories with definitions under this; INGCATEGORY andINGRECOVERYv INGCATEGORY, you need to modify to add categorization rules for your

address spaces.v INGRECOVERY, use appropriate actions for the categorization.

You may want to inherit definitions for all LOOPSUPP monitors using theC_LOOPSUPP APL class.

Adding a category rule

Under the MESSAGES/USER DATA Policy, place a K against INGCATEGORY andpress Enter.



|

|||

||

|

|||

|

|

|

|

||||

|

|

|

||||

||

|

||

||

|

||

|

||

Enter an I against the line at the bottom (SYSOTHER if you using an unmodifiedsample) and press Enter.

Now the codes you can enter on this panel are:

Code Description

CODE1 The address spaces WLM Service Class.

CODE2 The address spaces job type: STC, BATCH, TSO. This value willbe suffixed with _SA as the address space defined to SA z/OSas an APL.

CODE3 The value of jobname.stepname for the address space. Note thatthe code field is only 15 characters long, so you must use awildcard if you want to make the check critical upon the finalcharacters of the step name.

The output of the code match is the recovery category and this defaults to thename of the WLM Service Class.

The sample policy puts all resources known to SA z/OS and all SYSOTHERresources into the WARN recovery class, and all TSO users into the TSO_USERrecovery category.

Press PF3 to exit when you are done.

Adding a Recovery Sequence

If you would like a recovery sequence to be generally available for all of yourlooping address space monitors, then you should add it to the C_LOOPSUPP APLclass. If you add a recovery sequence to an APL you break the link between it andthe C_LOOPSUPP INGRECOVERY data, so it does not pick up any furtherchanges made to the class.

The codes passed into the code match are as follows:

Code Description

CODE1 The name of the Recovery Category; result of INGCATEGORY.

CODE2 The current recovery pass for the address space. The recoverypass is reset if there is a monitor cycle where the address spaceis not reported as looping or if the monitor APL is stopped.This is normally a number, but if you specify an * the reactionhappens on every pass.

CODE3 <blank>

The output is a list of one or more recovery actions:

Action Description

NONE Do nothing.



||

|

|||

||

||||

||||||||

|||

|

|

|||||

|

|||

||

||||||

|||

|

|||

||

Action Description

WARN Issue a warning message, ING601E. The message is written tothe Netlog and notify operators setup to be informed on behalfof class 40 and 44 to receive the warning message.

Warning messages are posted to SDF under the MVSESAresource. The number of passes the job has been found ondetermines their severity:

<10 -> UNUSUAL

<20 -> IMPORTANT

20 and higher -> CRITICAL

The messages are cleared when the resource is no longerdiscovered or when the monitor is stopped.

DIAG Use the OMEGAMON Inspection tool to obtain diagnostics onthe address space and write a diagnostic report to the Netlog.This can assist in identifying where in the code the loop isoccurring.

STOP Issue an MVS STOP command for the address space, or, if it isa TSO USER, an MVS CANCEL USER command is issued. Ifused with an address space defined to SA z/OS the STOPcommand will still be issued, but it will not be translated intoan INGREQ STOP request. This is to allow you to code a directshutdown sequence on a pass after you have tried a SHUT_command.

CANCEL Issue a MVS CANCEL command for the address space or, if itis a TSO USER, an MVS CANCEL USER command is issued. Ifused with an address space defined to SA z/OS the CANCELcommand will still be issued, but it will not be translated intoan INGREQ STOP request. This is to allow you to code a directshutdown sequence on a pass after you have tried aSHUT_command

FORCE Issue an MVS FORCE, ARM command for the address space,or if it is a TSO USER, an MVS CANCEL USER command isissued. If used with an address space defined to SA z/OS theFORCE command is still issued, but it is not translated into anINGREQ STOP request. This is to allow you to code a directshutdown sequence on a pass after you have tried a SHUT_command.

SHUT_NORM Issue an INGREQ STOP for shutdown type NORM for anaddress space defined to SA z/OS. If used with an addressspace not defined to SA z/OS, it is translated into a STOPinstruction.

SHUT_IMMED Issue an INGREQ STOP for shutdown IMMED for an addressspace defined to SA z/OS. If used with an address space notdefined SA z/OS, it is translated into a CANCEL instruction.

SHUT_FORCE Issue an INGREQ STOP for shutdown type FORCE for anaddress space defined to SA z/OS. If used with an addressspace not defined to SA z/OS, it is translated into a FORCEinstruction.

SUSPEND Issue an MVS RESET,QUIESCE for the address space. Thisstops any further CPU dispatching for it, but not stop it.



||

||||

|||

|

|

|

||

|||||

||||||||

||||||||

||||||||

|||||

||||

|||||

|||

Action Description

RESET_class Issue an MVS RESET,SRVCLASS=class for the address space.This changes its WLM dispatching class, preferably to onewhere it consumes less CPU. Changing the address space toclass SYSOTHER reduces its dispatching priority to a very lowvalue.

USER_prog The prog parameter is the name of a user program which, iffound, is invoked during the recovery process.

The parameters passed to the routine are:

asid jobname jobid jobstep cpuindex

The sample policies provide 4 recovery categories:

Recovery Category Description

IGNORE This makes the monitor leave the job alone and not tell anyoneabout it. It is designed to be used with known false positives.

WARN This outputs a warning message every pass and gathersdiagnostics on the second pass.

SUSPEND This outputs a warning message on every pass, gathersdiagnostics on the second pass and issues a SUSPEND for theaddress pace on the 4th pass. There is no reminder that theaddress space has been suspended. You need to check yourlogs.

TSO_USER This simply issues a warning message on every pass.

Again, press PF3 to exit when you are done.

LinkageNow that your categorization and recovery policies are defined (you can comeback later and refine them), you need to link the APL to one or more systemswhere it runs.

Note that the systems it gets linked to must have a Network (NTW) defining aSOAP server called HUBTEMS linked to them or the monitor does not work.

To do this, create an APG with a blank automation name, make yourLOOPSUPP_identity APL a member of it and link it to the systems where youwant the monitor to run. You can model the APG on the ING_ANALYTIC APG ifyou wish.

BuildYou now need to build your SOCNTL files and get them ready to be activated onthe target system.

Prime the CredentialsYou now need to issue a command to set up the password that the monitor uses toaccess the TEMS server.

Log onto the Automation Agent on each system that you are going to run themonitor on and issue the following commands:NETVASIS INGPW userid SOAP,INIT=password

so if your userid was AutoAgnt and your password was ABC1234d you would issue:



||

||||||

|||

|

|||

|||

|||

|||

||||||

||||

|

|||

||

||||

|

||

|

||

||

|

|

NETVASIS INGPW AutoAgnt SOAP,INIT=ABC1234d

When you subsequently change the password you need to reissue this commandto tell System Automation what the new password is.

Activate the MonitorIssue an INGAMS REFRESH to pick up the new SOCNTL files and the monitorstarts running.

Verify the InstallationWait 5 minutes and then check the netlog. Look for an ING600I message.

This is the monitoring report that the procedure produces. It should indicate thatthe query was successful and may or may not contain details of any loopingaddress spaces that were found.

AOFRSA01

PurposeYou can use the AOFRSA01 automation routine to respond to logrec data setnearly full or full messages from your system by issuing commands from theconfiguration files to dump and clear the contents of the logrec data set.

AOFRSA01 keeps track of the incoming logrec data set messages and comparestheir occurrence with predefined thresholds of infrequent, frequent, and criticallevel. An exceeded threshold is used as the option to select the appropriatecommands according to the first field in the command entry of theMVSESA/LOGREC entry/type-pair in the configuration file. If no threshold isexceeded the commands defined for the selection option ALWAYS are issued.

AOFRSA01 should be called from the NetView automation table.

Syntax

►► AOFRSA01 ►◄

Restrictionsv Actions are only taken in AOFRSA01 if the recovery automation flag for

LOGREC is on.v Processing in AOFRSA01 is only done if it is called from NetView automation

table by one of the expected messages IFB040I, IFB060E, IFB080E or IFB081I.v The commands from automation policy to dump and clear the LOGREC data set

are only issued if a LOGREC recovery function is not already in progress.

UsageAutomation routine AOFRSA01 is intended to respond to the following messages:IFB040I SYS1.LOGREC AREA IS FULL, hh.mm.ssIFB060E SYS1.LOGREC NEAR FULLIFB080E LOGREC DATA SET NEAR FULL, DSN=dsnameIFB081I LOGREC DATA SET IS FULL,hh.mm.ss, DSN=dsn



|

||

|

||

|

|

|||

The commands to issue are selected from the command entry of theMVSESA/LOGREC entry/type-pair in the configuration file.

If no threshold is reached when one of the expected messages arrive, allcommands to entries with no selection option and to selection option ALWAYS areselected. If the threshold at level infrequent is exceeded, all commands to entrieswith no selection specification option and to selection option INFR are selected. Inthe same way a level of frequent corresponds to selection option FREQ and a levelof critical corresponds to selection option CRIT.

Make sure that the automation routine AOFRSA02 is issued by message IFC001Ifrom the NetView automation table, to indicate the completion of the LOGRECrecovery function.

Global Variables&EHKVAR1

When defining the commands in the configuration files to dump and clearthe contents of the LOGREC data set, the variable &EHKVAR1 can be usedfor the name of the LOGREC data set. This variable is substituted with thecomplete data set name of the LOGREC data set name.

AOFRSA02

PurposeYou can use the AOFRSA02 automation routine to respond to the initializationmessage of the LOGREC data set to reset the flag, which indicates that theLOGREC recovery function is in progress


Syntax


Restrictionsv Actions are only taken in AOFRSA02 if the recovery automation flag for

LOGREC is on.v Processing in AOFRSA02 is only done if it is called from NetView automation

table.

UsageAutomation routine AOFRSA02 is intended to respond to the following message:IFC001I D=devtyp N=x F=track1* L=track2* S=recd** DIP COMPLETE

This is produced during the initialization of the LOGREC data set and describesthe limits of the data set.

The flag, indicating that the LOGREC recovery function is in progress, is used byautomation routine AOFRSA01.

AOFRSA01


ExamplesThis example shows a sample scenario for LOGREC data set processing:

The following entries in the NetView automation table are created automatically toissue the appropriate automation routine when one of the expected messagesarrives:IF MSGID = ’IFB040I’ | MSGID = ’IFB060E’ |

MSGID = ’IFB080I’ | MSGID = ’IFB081I’THENEXEC(CMD(’AOFRSA01’)ROUTE(ONE %AOFOPRECOPER%));

IF MSGID = ’IFC001I’THENEXEC(CMD(’AOFRSA02’)ROUTE(ONE %AOFOPRECOPER%));

Assume that the following message arrives the first time for one day:IFB080E LOGREC DATA SET NEAR FULL, DSN=SYS1.AOC1.MAN3

Because none of the defined thresholds is exceeded, the automation routineAOFRSA01 searches for defined commands without selection option and toselection option ALWAYS to be issued. With the control file shown above thecommand MVS S CLRLOG,DSN=&EHKVAR1 is selected. Before issuing this command,the variable &EHKVAR1 is substituted by the data set name of the receivedmessage resulting in MVS S CLRLOG,DSN=SYS1.AOC1.MAN3.

If message IFB080E continues to arrive and the occurrence of the expectedmessages thus exceeds the infrequent, frequent or critical threshold, the automationroutine AOFRSA01 searches for defined commands without selection option and toselection option INFR, FREQ or CRIT to be issued.

Because no command is defined with any selection option, only the definedcommand with no selection option is selected and issued, as in the previous case.

COMMANDS HELP------------------------------------------------------------------------------

Thresholds DefinitionCommand ===>

Entry Type : MVS Component PolicyDB Name : DATABASE_NAMEEntry Name : MVS_COMPONENTS Enterprise Name : YOUR_ENTERPRISE

Resource : MVSESA.LOGREC

Critical Number . . . . 3 (1 to 50)Critical Interval . . . 00:05 (hh:mm or hhmm, 00:01 to 24:00)

Frequent Number . . . . 3 (1 to 50)Frequent Interval . . . 00:30 (hh:mm or hhmm, 00:01 to 24:00)

Infrequent Number . . . 3 (1 to 50)Infrequent Interval . . 24:00 (hh:mm or hhmm, 00:01 to 24:00)

Figure 33. Threshold Definitions for MVS Component LOGREC

Cmd PS/Select AutoFn/* Command TextMVS S CLRLOG,DSN=&EHKVAR1

Figure 34. MESSAGES/USER DATA Policy Item for Entry/Type-Pair MVSESA/LOGREC

AOFRSA02


|||||||

Message AOF589I, AOF588I or AOF587I is issued in cases, where an infrequent,frequent or critical threshold has been exceeded. These messages indicate that aninfrequent, frequent or critical threshold action has been processed.

If the recovery processing for a LOGREC data set is still in progress when anexpected error message arrives, the following message is issued:AOF585I 15:45 : RECOVERY OF LOGREC IS ALREADY IN PROGRESS -

The recovery process is considered to be finished, when message IFC001I arrivestelling that the LOGREC data set has been initialized.

AOFRSA03

PurposeYou can use the AOFRSA03 automation routine to respond to SMF data set full orswitch messages from your system. AOFRSA03 issues commands from theconfiguration files to dump and clear the contents of the SMF data set.

AOFRSA03 keeps track of incoming SMF data set messages and compares theiroccurrence with predefined thresholds at infrequent, frequent, and critical levels.An exceeded threshold is used as the option for selecting the appropriatecommands according to the first field in the command entry of theMVSESA/SMFDUMP entry/type pair in the configuration file. If no threshold isexceeded the commands that are defined for the selection option ALWAYS areissued.


Besides that kind of automation, SA z/OS also checks for full SMF data sets whichwere filled up while SA z/OS was not active. For each data set where a dump isrequired a command is issued if the selection option is set to 'ALWAYS'.&EHKVAR contains fully qualified SMF data set name.

Syntax


Restrictionsv Processing in AOFRASA03 is done if it is called from NetView automation table

by one of the expected messages: IEE362A, IEE262I, IEE391A or IEE392I.v If AOFRASA03 was triggered by one of the above messages, then actions are

only taken if the recovery automation flag for SMFDUMP is on.v Actions in AOFRSA03 are only taken if the recovery automation flag for

SMFDUMP is on.v Processing in AOFRSA03 is only done if it is called from the NetView

automation table by one of the expected messages: IEE362A, IEE262I, IEE391A orIEE392I.

UsageAutomation routine AOFRSA03 is intended to respond to the following messages:

AOFRSA02


IEE362A SMF ENTER DUMP FOR dsname ON serIEE362I SMF ENTER DUMP FOR dsname ON serIEE391A SMF ENTER DUMP FOR DATA SET ON VOLSER ser, DSN=dsnameIEE392I SMF ENTER DUMP FOR DATA SET ON VOLSER ser, DSN=dsname

that indicate that the SMF data set is ready to be dumped.

Global Variables&EHKVAR1

When defining the commands in the configuration file to dump and clearthe contents of the SMF data set, the variable &EHKVAR1 can be used forthe name of the SMF data set. This variable is substituted with thecomplete data set name by AOFRSA03 when message IEE391A or IEE392Iis received. In case of message IEE362A or IEE362I this variable issubstituted with MANn, the second part of the SMF data set name.

&EHKVAR2When defining the commands in the configuration file to dump and clearthe contents of the SMF data set, the variable &EHKVAR2 can be used forthe name of the SMF data set. This variable is substituted with thecomplete data set name by AOFRSA03 when message IEE391A, IEE392I,IEE362A, or IEE362I is received.

ExamplesThis example shows SMF data set processing when AOFRSA03 is called from theautomation table.

The following entries in the NetView automation table are created automatically toissue the appropriate automation routine when one of the expected messagesarrives:IF (MSGID = ’IEE362I’ | MSGID = ’IEE362A’ |

MSGID = ’IEE391A’ | MSGID = ’IEE392I’)THENEXEC(CMD(’AOFRSA03’)ROUTE(ONE %AOFOPRECOPER%));

COMMANDS HELP------------------------------------------------------------------------------



Resource : MVSESA.SMFDUMP




Figure 35. Threshold Definitions for MVS Component SMFDUMP

AOFRSA03


Assume that the following message arrives the first time on one day:IEE391A SMF ENTER DUMP FOR DATASET ON VOLSER 123, DSN=SYS1.AOC1.MAN3

Because none of the defined thresholds has been exceeded, the AOFRSA03automation routine searches for commands to issue that have been defined withouta selection option or with the selection option ALWAYS. With the control fileshown above the command MVS S SMFDUMP1,DA=’&EHKVAR2’ is selected. Beforeissuing this command, the variable &EHKVAR2 is substituted with the data setname from the received message, resulting in MVS SSMFDUMP1,DA=’SYS1.AOC1.MAN3’.

If message IEE391A continues to arrive and the occurrence of the expectedmessages thus exceeds the infrequent, frequent or critical thresholds, theAOFRSA03 automation routine searches for commands to issue that have beendefined without a selection option or with selection option INFR, FREQ or CRIT.

Because no command has been defined with a selection option, only the commandthat has been defined without a selection option is selected and issued, as in theprevious case.

Message AOF589I, AOF588I or AOF587I is issued in cases where an infrequent,frequent or critical threshold has been exceeded. These messages indicate that aninfrequent, frequent or critical threshold action has been processed.

AOFRSA08

PurposeYou can use the AOFRSA08 automation routine to respond to syslog being queuedmessages by starting an external writer to save the syslog that was queued.

AOFRSA08 keeps track of the incoming syslog queued messages and comparesthere occurrence with predefined thresholds at infrequent, frequent, and criticallevels. An exceeded threshold is used as the option for selecting the appropriatecommands according to the first field in the command entry of theMVSESA/SYSLOG entry/type-pair in the configuration file. If no threshold isexceeded the commands that are defined for the selection option ALWAYS areissued.


Syntax


Restrictionsv Processing in AOFRSA08 is only done if it is called from NetView automation

table by the expected message IEE043I.

Cmd PS/Select AutoFn/* Command TextMVS S SMFDUMP1,DA=’&EHKVAR2’

Figure 36. MESSAGES/USER DATA Policy Item for Entry/Type-Pair MVSESA/SMFDUMP

AOFRSA03


|||

v Actions are only taken in AOFRSA08 if the recovery automation flag forSYSLOG is on and if the status of JES is UP or HALTED.

UsageAutomation routine AOFRSA08 is intended to respond to the message:IEE043I A SYSTEM LOG DATA SET HAS BEEN QUEUED TO SYSOUT CLASS class

which indicates that the system closed the system log (SYSLOG) data set andqueued the data set to a SYSOUT class.

The commands to issue are selected from the command entry of theMVSESA/SYSLOG entry/type-pair in the configuration file.

If no threshold is reached when one of the expected messages arrive, allcommands that are defined for entries without a selection option and for theselection option ALWAYS are selected.

If the threshold at the infrequent level is exceeded, all commands that are definedfor entries without a selection specification option and for entries with the selectionoption INFR are selected.

In the same way, a level of frequent corresponds to the selection option FREQ anda level of critical corresponds to the selection option CRIT.

ExamplesThis example shows a sample scenario for SYSLOG processing:

The following entry in the NetView automation table is created automatically toissue AOFRSA08 as response to the incoming IEE043I message:IF MSGID = ’IEE043I’THENEXEC(CMD(’AOFRSA08’)ROUTE(ONE %AOFOPRECOPER%));

COMMANDS HELP------------------------------------------------------------------------------



Resource : MVSESA.SYSLOG




Figure 37. Threshold Definitions for MVS Component SYSLOG

AOFRSA08


Assume that the following message arrives the first time for one day:IEE043I A SYSTEM LOG DATA SET HAS BEEN QUEUED TO SYSOUT CLASS A

Because none of the defined thresholds is exceeded, the automation routineAOFRSA08 searches for defined commands without selection option and toselection option ALWAYS to be issued. With the control file shown above thecommand MVS S SAVELOG is selected.

If message IEE043I continues to arrive and the occurrence of the expectedmessages thus exceeds the infrequent, frequent or critical threshold, the automationroutine AOFRSA08 searches for defined commands without selection option and toselection option INFR, FREQ or CRIT to be issued.

Because no command is defined with any selection option, only the definedcommand with no selection option is selected and issued, as in the previous case.

Message AOF589I, AOF588I or AOF587I is issued in cases, where an infrequent,frequent or critical threshold has been exceeded. These messages indicate that aninfrequent, frequent or critical threshold action has been processed.

AOFRSA0C

PurposeYou can use the AOFRSA0C automation routine to respond to a SVC dump takento a dump data set message by issuing commands from the configuration file toformat the dump, to clear the dump data sets, or to prevent further dumping. Thecommands to issue are taken from the MVSESA/MVSDUMP andMVSESA/MVSDUMPTAKEN entry/type-pairs and selected according to thefrequency of the incoming messages and the thresholds defined in the automationpolicies. The first field in the command entry gives detailed criteria to select theappropriate commands from the configuration file.

AOFRSA0C should be called from the NetView automation table.

Syntax

►► AOFRSA0C ►◄

Restrictionsv Actions in AOFRSA0C are only taken if the recovery automation flag for

MVSDUMP is on.v Processing in AOFRSA0C is only done if it is called from NetView automation

table by one of the expected messages IEA611I or IEA911E.

UsageAutomation routine AOFRSA0C is intended to respond to the following messages:

Cmd PS/Select AutoFn/* Command TextMVS S SAVELOG

Figure 38. MESSAGES/USER DATA Policy Item for Entry/Type-Pair MVSESA/SYSLOG

AOFRSA08


IEA611I {COMPLETE|PARTIAL} DUMP ON dsnameDUMPID=dumpid REQUESTED BY JOB (jobname)FOR ASIDS(id,id,...)...

IEA911E {COMPLETE|PARTIAL} DUMP ON SYS1.DUMPnnDUMPid=dumpid REQUESTED BY JOB (jobname)FOR ASIDS(id,id,...)...

These indicate that the system wrote a complete or partial SVC dump to anautomatically allocated or pre-allocated dump data set on a direct access storagedevice or a tape volume.

AOFRSA0C keeps track on the reception of these messages and compares thefrequency of the incoming messages with predefined thresholds of infrequent,frequent and critical level, where the thresholds to MVS component MVSDUMPare considered. The commands to issue are selected according to the frequency ofthe incoming messages.

If no threshold is reached, all commands to entries with no selection option and toselection option ALWAYS are selected. If the threshold at level infrequent isexceeded, all commands to entries with no selection option and to selection optionINFR are selected. In the same way a level of frequent corresponds to selectionoption FREQ and a level of critical corresponds to selection option CRIT.

The commands to issue are taken from MVSESA/MVSDUMP entry/type-pair ofthe configuration file with respect to the frequency of the incoming of thesemessages.

If AOFRSA0C has been triggered by receipt of message IEA911E, all the commandsfrom the MVSESA/MVSDUMPTAKEN entry/type-pair of the configuration file arealso selected and issued, as long as the critical threshold has not been exceeded.

After dump processing has been done, AOFRSA0C further monitors the frequencyof messages IEF611I and IEF911E in intervals of 15 minutes. As soon as thefrequency falls below the infrequent threshold, all the commands ofMVSESA/MVSDUMPRESET entry/type-pair are issued.

Global VariablesWhen defining the commands in the configuration file to handle the SVC dumpdata set, the variables &EHKVAR1 to &EHKVAR6 can be used to be substituted byvariable contents of message IEA611I or IEA911E. The variables &EHKVAR1 to&EHKVAR6 are not available in command entries of type MVSDUMPRESET.These variables are substituted as follows:

&EHKVAR1The dsname of IEA611I or suffix of SYS1.DUMPnn in IEA911E

&EHKVAR2The data set name

&EHKVAR3The dump ID

&EHKVAR4The job name

AOFRSA0C


&EHKVAR5The ID of address space

&EHKVAR6The dump type (PARTIAL or COMPLETE)

ExamplesThis example shows the use of automation routine AOFRSA0C in a samplecontext:

An entry in the NetView automation table is used to issue AOFRSA0C when oneof the expected messages arrives:IF MSGID = ’IEA611I’ | MSGID = ’IEA911E’THENEXEC(CMD(’AOFRSA0C ’)ROUTE(ONE %AOFOPRECOPER%));

Three threshold levels are defined in the automation policy for MVS componentMVSDUMP:

The MESSAGES/USER DATA automation policy item of the MVSESA/MVSDUMPentry/type-pair contains the following command entries for the message IDMVSDUMP with selection options at different levels:

Ps/Select Command Text

FREQ ’MVS DD ALLOC=INACTIVE’

INFR ’MVS DD ALLOC=ACTIVE’

CRIT ’MVS DD ALLOC=INACTIVE’

The MESSAGES/USER DATA automation policy item of the MVSESA/MVSDUMPTAKEN entry/type-pair contains the following entry without anyselection options:’MVS DD CLEAR,DSN=&EHKVAR1’

AOFKAASR SA z/OS - Command DialogsDomain ID = IPSNO ---------- INGTHRES ---------- Date = 08/28/03Operator ID = SAUSER Time = 09:38:02

Specify thresholds and resource changes:

Resource => MVSESA.MVSDUMP Group or specific resourceSystem => KEY3 System name, domain ID, sysplex name or *all

Critical => 6 errors in 00:30 Time (HH:MM)Frequent => 4 errors in 00:20 Time (HH:MM)Infrequent => 2 errors in 00:20 Time (HH:MM)

Pressing ENTER will set the THRESHOLD values

Command ===>PF1=Help PF2=End PF3=Return PF6=Roll

PF12=Retrieve

Figure 39. MVSDUMP Thresholds

AOFRSA0C


The MESSAGES/USER DATA automation policy item of the MVSESA/MVSDUMPRESET entry/type-pair contains the following entry without anyselection options:’MVS DD ALLOC=ACTIVE’

As long as no threshold is exceeded at receipt of one of the IEA611I and IEA911Emessages, no action is taken.

If dumps have been taken more often than defined with the infrequent threshold,the command MVS DD ALLOC=ACTIVATE, specified in entry type MVSDUMP, isissued. This makes sure that automatic dump data set allocation is enabled. Incases when the dump has been written to a pre-allocated SYS1.DUMP data set,additionally the data set is cleared using the command MVS DDCLEAR,DSN=&EHKVAR1, specified in the entry type MVSDUMPTAKEN. Variable&EHKVAR1 is substituted by the numeric suffix of the SYS1.DUMP data set.

The same processing is done in cases when the incoming dump data set messagesexceeds the frequent level.

As soon as the critical threshold is exceeded, the automation routine stops clearingpre-allocated SYS1.DUMP data sets.

After commands having been issued by the automatic processing of dump datasets, automation routine AOFRSA0C checks every 15 minutes whether theinfrequent threshold is satisfied again. As soon as this situation is reached,automatic dump data set allocation is enabled again by command MVS DDALLOC=ACTIVE, as defined in entry type MVSDUMPRESET.

AOFRSA0E

PurposeAutomation routine AOFRSA0E deletes WTORs from SA z/OS display capabilitieswhen they are replied to or canceled.

Syntax

►► AOFRSA0E ▼

,

id►◄

Parametersid The reply identifiers for cancelled messages.

RestrictionsProcessing in AOFRSA0E is only done if it is called from NetView automationtable by message IEE400I or IEE600I or if one of these messages are passed byparameter.

UsageAutomation routine AOFRSA0E is intended to respond to the following messages:

AOFRSA0C


IEE400I THESE MESSAGES CANCELED- id,id,idIEE600I REPLY TO id IS; text

Message IEE400I says that the system cancelled messages because the issuing taskended or specifically requested that the messages be cancelled. Message IEE600Inotifies all consoles that received a message that the system accepted a reply to themessage.

As well AOFRSA0E can extract the identifiers of the messages to delete frompassed parameters.

ExampleThe following example shows how to issue AOFRSA0E from the NetViewautomation table:IF MSGID = ’IEE400I’ | MSGID = ’IEE600I’THENEXEC(CMD(’AOFRSA0E ’)ROUTE(ONE %AOFOPWTORS%));

AOFRSA0G

PurposeYou can use the AOFRSA0G automation routine to respond to messages reportingbuffer shortage of the action message retention facility (AMRF) by issuingcommands from the configuration file to process buffer shortage automation.

In the case of an incoming buffer shortage message, the commands to issue aretaken from the MVSESA/AMRFSHORT entry/type-pair with the selection optionPASS1 and reissued at 1 minute intervals with an incremented pass count.

In the case of a buffer full message, the commands to issue are taken from theMVSESA/AMRFFULL entry/type-pair. If buffer shortage relieved is reported, thecommands that are defined for the MVSESA/AMRFCLEAR entry/type-pair areselected.

AOFRSA0G should be called from the NetView automation table.

Syntax

►► AOFRSA0G ►◄

Restrictionsv Actions are only taken in AOFRSA0G if the recovery automation flag for AMRF

is on.v Processing of system messages in AOFRSA0G is only done if it is called from

NetView automation table by message IEA359E, IEA360A or IEA361I.

UsageAutomation routine AOFRSA0G is intended to respond to the messages:IEA359E BUFFER SHORTAGE FOR RETAINED ACTION MESSAGES - 80% FULLIEA360A SEVERE BUFFER SHORTAGE FOR RETAINED ACTION MESSAGES - 100% FULLIEA361I BUFFER SHORTAGE RELIEVED FOR RETAINED ACTION MESSAGES

AOFRSA0E


IEA359E and IEA360A reports buffer shortage of the buffer area for immediateaction messages, non-critical and critical eventual action messages and WTORmessages. IEA361I indicates the reduction of the number of retained actionmessages so that the buffer is now less than 75% full.

If AOFRSA0G has been triggered on receipt of message IEA359E the commands toissue are taken from entry/type-pair MVSESA/AMRFSHORT, starting at selectionoption PASS1 and continuing with incremented selection options in 1 minuteintervals until message IEA361 reports that buffer shortage has relieved. Afterarriving the maximal used selection option for a defined command processingrestarts at selection option PASS1.

If AOFRSA0G has been triggered on receipt of message IEA360A all commandsfrom entry/type-pair MVSESA/AMRFFULL are issued.

If AOFRSA0G has been triggered on receipt of message IEA361I all commandsfrom entry/type-pair MVSESA/AMRFCLEAR are issued.

ExamplesThe following example shows a sample scenario for AMRF shortage processing:

Entries in the NetView automation table are used to issue AOFRSA0G whenmessage IEA359E, IEA360E or IEA361I arrives:IF MSGID = ’IEA359E’THENEXEC(CMD(’AOFRSA0G’)ROUTE(ONE %AOFOPRECOPER%));IF MSGID = ’IEA360A’THENEXEC(CMD(’AOFRSA0G’)ROUTE(ONE %AOFOPRECOPER%));IF MSGID = ’IEA361I’THENEXEC(CMD(’AOFRSA0G’)ROUTE(ONE %AOFOPRECOPER%));

To specify how to respond to message IEA359E and IEA361I, the followingcommand definitions are made in the automation policy under the entry/type-pairMVSESA/AMRFFULL and MVSESA/AMRFCLEAR:

If for example messageIEA360A SEVERE BUFFER SHORTAGE FOR RETAINED ACTION MESSAGES - 100% FULL

arrives, AOFRSA0G is issued by the shown statement in the NetView automationtable, which causes the command CONTROL M,AMRF=N to be issued to clear theAMRF buffers.

After AMRF buffer shortage is relieved, the incoming message

Command = ACF ENTRY=MVSESA,TYPE=AMRF*,REQ=DISPSYSTEM = AOC1 AUTOMATION CONFIGURATION DISPLAY - ENTRY= MVSESA-------------------------------------------------------------------------------AUTOMATION CONFIGURATION DISPLAY - ENTRY= MVSESATYPE IS AMRFCLEARCMD = (,,’MVS CONTROL M,AMRF=Y’)TYPE IS AMRFFULLCMD = (,,’MVS CONTROL M,AMRF=N’)END OF MULTI-LINE MESSAGE GROUP

Figure 40. MVSESA AMRF Command Definitions

AOFRSA0G


|||||||||

IEA361I BUFFER SHORTAGE RELIEVED FOR RETAINED ACTION MESSAGES

causes command CONTROL M,AMRF=Y to be issued to reactivate AMRF.

AOFRSD07

PurposeYou can use the AOFRSD07 automation routine to respond to a JES2 not dormantmessage during JES2 shutdown by issuing commands for resources that are notdrained.

The commands to issue are taken from the automation policy item JES2 DRAIN ofapplication JES2.

Additionally AOFRSD07 calls AOFRSD0F which outputs a list of all active jobs andstarted tasks and a list of all resources not yet drained.

AOFRSD07 should be called from the NetView automation table.

Syntax

►► AOFRSD07 ►◄

RestrictionsProcessing in AOFRSD07 is only done if:v It is called from NetView automation table by JES2 message HASP607v The terminate automation flag for JES2 is onv JES2 is in shutdown progress

UsageAutomation routine AOFRSD07 is intended to respond to messageHASP607 JES2 NOT DORMANT -- MEMBER DRAINING, RC=rc text

which indicates in case the P JES2 command was entered to withdraw JES2 fromthe system that not all of JES2's functions have completed.

To find out all resources not drained, the response to JES2 command DU,STA isprocessed. For each resource in status DRAINING the corresponding commandfrom the automation policy item JES2 DRAIN for this resource type to force drainis issued. Resources in status ACTIVE are first stopped with JES2 command Presource, before the command from the automation policy item to force drain isissued. Resources in status INACTIVE are only stopped with JES2 command Presource.

In cases, where the automation is unable to issue actions on not yet drainedresources, JES2 is set to status STUCK and a message is issued which tells that anoperator action is required. Those situations occur if no command is specified inautomation policy item JES2 DRAINED of JES2 to drain a resource or if a not yetdrained resource is in an unknown status

AOFRSA0G


AOFRSD09

PurposeAutomation routine AOFRSD09 is used for JES2 spool recovery. It is called byAOFRSD01 via a timer every retry interval to monitor spool utilization of JES2 andto successive issue the recovery commands of policy item JES2 SPOOLSHORT orJES2 SPOOLFULL.

For this purpose AOFRSD09 processes the following steps:v AOFRSD09 issues the JES2 command D SPOOL to obtain the current spool

usage.v AOFRSD09 re-evaluates the target of recovery process based on the actual

warning threshold for TG and the buffer value from the configuration file.v If the recovery target has not yet been achieved and the current JES2 subsystem

is responsible for the spool recovery, AOFRSD09 increments the pass count andissues the appropriate commands from the configuration file. In a shared JES2environment, where all JES2 subsystems receive a copy of the spool shortagemessage, AOFRD09 determine the appropriate JES2 subsystem for spoolrecovery. To do this, AOFRD09 compares the list of cpuids, as defined inconfiguration file, with the response to JES2 command DMEMBER,STATUS=ACTIVE. The first active cpuid on the list is considered to bethe appropriate JES2 subsystem for spool recovery.

v In case the spool shortage problem has already been relieved, AOFRSD09 stopsthe recovery process and sets a timer to reset the pass count for the recoverycommands after the reset interval.

You define recovery commands and configuration parameters for JES2 recoveryprocessing, such as buffer value, reset interval and cpuid list, using automationpolicy item JES2 SPOOLSHORT for spool shortage recovery processing and JES2SPOOLFULL for spool full recovery processing.

For further information about the JES2 SPOOLSHORT and JES2 SPOOLFULLautomation policy items see IBM Tivoli System Automation for z/OS DefiningAutomation Policy.

Syntax

►► AOFRSD09 subsystem recovery type ►◄

Parameterssubsystem

The subsystem name of JES2. This parameter is required.

recovery typeThis parameter is used to distinguish between a JES2 spool shortage and aJES2 spool full condition. This parameter is required.

SHORTThe automatic recovery from a JES2 spool shortage condition is to beprocessed.

FULL The automatic recovery from a JES2 spool full condition is to beprocessed.

AOFRSD09


Restrictionsv Processing of recovery commands in AOFRSD09 is only done if the recovery

automation flag for JES2 is on. Otherwise the recovery process is suspended andthe pass count for selection recovery commands from the configuration file isnot incremented.

v Automation routine AOFRSD09 should be processed by JESOPER. If it is calledon another task it is routed back to JESOPER.

v Processing in AOFRSD09 is only done if the specified type of spool recoveryprocess has been initiated by automation routine AOFRSD01.

v During a SPOOLFULL recovery condition, the processing for SPOOLSHORTrecovery is suspended.

UsageThe recovery commands to issue are selected from the command entry of policyitem JES2 SPOOLSHORT or JES2 SPOOLFULL. A pass count is used as selectionoption and incremented at each successive processing of automation routineAOFRSD09. At initialization of the recovery process, the pass count is set to valuePASS1 by automation routine AOFRSD01.

If pass processing runs out of defined recovery commands before the spoolshortage condition is resolved, AOFRSD09 re-executes the recovery sequence fromPASS1. You can change this behavior by setting the appropriate advancedautomation option at start up of System Automation. You can use theAOFSPOOLSHORTCMD variable (for SPOOLSHORT conditions) and theAOFSPOOLFULLCMD variable (for SPOOLFULL conditions) to tell automationroutine AOFRSD09 to stop recovery attempts when all commands have beenexecuted and to issue message AOF294I to inform the operator that manualintervention is required in order to resolve the spool condition. For moreinformation about advanced automation options refer to “Read/Write Variables”on page 242.

Global VariablesWhen defining the commands in the SPOOLFULL or SPOOLSHORT processingpanel of the configuration file to handle the recovery, the variables &EHKVAR1and &EHKVAR2 can be used to be substituted by variable contents. Variable&EHKVAR1 is substituted by the current spool utilization and &EHKVAR2contains the recovery target.

AOFRSD0F

PurposeAutomation routine AOFRSD0F is used by AOFRSD07 for drain processing priorto JES2 shutdown. Every shutdown delay interval, AOFRSD0F displays all JES2resources not yet drained. For this purpose it scans the response to JES2 commandDA,S for executing tasks, the response to JES2 command DA,J for executing jobsand the response to JES2 command DU,STA for started devices or lines not yetdrained and displays the result in a message.

Syntax

►► AOFRSD0F subsystem ►◄

AOFRSD09


Parameterssubsystem

The subsystem name of JES2.

RestrictionsProcessing in AOFRSD0F is only done if the following conditions are met:v The subsystem is of type JES2v JES2 is in shutdown progressv The terminate automation flag is on

UsageThis automation routine is performed as part of the SHUTDOWN processing.

ExamplesThis example shows a sample scenario for JES2 drain processing prior to JES2shutdown.

The following statement shows how AOFRSD07 is issued from the NetViewautomation table by JES2 message$HASP607: IF MSGID(2) = ’HASP607’THENEXEC(CMD(’AOFRSD07’)ROUTE(ONE %AOFOPJESOPER%));

Assume the following drain processing specifications in automation policy itemJES2 DRAIN:

The list of commands to force drain of JES2 resources are passed to theJES2/FORCEDRAIN entry/type-pair in the configuration file and can be displayedwith the DISPACF command:

COMMANDS HELP------------------------------------------------------------------------------

JES2 DRAIN Specification Line 00000001Command ===> Scroll ===> PAGE

Entry Type : Application PolicyDB Name : DATABASE_NAMEEntry Name : JES2 Enterprise Name : YOUR_ENTERPRISE

Enter YES or NO for initial drain to bring down JES2 facilities:

Drain lines . . . . . . . . . . YES (YES NO)Drain VTAM interface . . . . . . YES (YES NO)Drain spool offloaders . . . . . NO (YES NO)Drain printers . . . . . . . . . YES (YES NO)Drain readers . . . . . . . . . YES (YES NO)Drain punches . . . . . . . . . YES (YES NO)

Enter NO or a command for force drain if normal drain fails:Force drain lines . . . . . . . $E (NO, Command)Force drain VTAM interface . . . $E (NO, Command)Force drain spool offloaders . . NO (NO, Command)Force drain printers . . . . . . $I (NO, Command)Force drain readers. . . . . . . $C (NO, Command)Force drain punches. . . . . . . $E (NO, Command)

Figure 41. JES2 DRAIN Specifications Panel

AOFRSD0F


||||||||||||||||||||||||||||

Assume that during a shutdown of JES2 message $HASP607 arrives, indicatingthat not all of JES2's functions have completed and that JES2's response tocommand $DU,STATUS is:$HASP636 13.53.22 $DU,STALINE1 UNIT=0FF3,STATUS=ACTIVE/BOEVM9,DISCON=NO

Automation routine AOFRSD07 first issues JES2 command $PLINE1 to stop the lineand then issues JES2 command $E, according to the policy specifications FORentry/type-pair JES2/FORCEDRAIN.

Then automation routine AOFRSD0F is executed every shutdown delay interval, tolist all JES2 resources not drained.

AOFRSD0G

PurposeYou can use the AOFRSD0G automation routine to drain JES2 resources prior toJES2 shutdown. AOFRSD0G issues commands to drain the initiators, offloadertasks, lines, printers, punches and readers, depending on which resources are listedand enabled in the automation policy item JES2 DRAIN of application JES2.

AOFRSD0G is used by the DRAINJES command.

Syntax

►► AOFRSD0G subsystem ►◄

Parameterssubsystem

The subsystem name of JES2.

RestrictionsProcessing in AOFRSD0G is only done if the subsystem is of type JES2.

UsageFor all resources enabled to initial drain in automation policy item JES2 DRAIN ofapplication JES2 the JES2 command P is issued.

Command = ACF ENTRY=JES2,TYPE=FORCEDRAIN,REQ=DISPSYSTEM = KEY3 AUTOMATION CONFIGURATION DISPLAY - ENTRY= JES2-------------------------------------------------------------------------------AUTOMATION CONFIGURATION DISPLAY - ENTRY= JES2TYPE IS FORCEDRAINLIN = ""$E""LOG = ""$E""OFF = ""NO""PRT = ""$I""RDR = ""$C""PUN = ""$E""END OF MULTI-LINE MESSAGE GROUP

Figure 42. DISPACF Panel

AOFRSD0F


ExampleCall AOFRSD0G JES2 to stop all resources enabled in JES2 DRAIN for init drain.

These resources can be listed with command DISPACF JES2 INITDRAIN.

AOFRSD0H

PurposeThe AOFRSD0H automation routine is used for JES2 spool recovery. It is called byAOFRSD09 with a timer command after the reset interval and cleans up the passcounter for the pass processing of the recovery commands of the configuration file.

Syntax

►► AOFRSD0H subsystem recovery type ►◄

Parameterssubsystem

The subsystem name of JES2. This parameter is required.

recovery typeThis parameter is used to distinguish between a JES2 spool shortage and aJES2 spool full condition. This parameter is required.

SHORTThe pass counter for spool shortage recovery processing is to be reset.

FULL The pass counter for spool full recovery processing is to be reset.

Restrictionsv The AOFRSD0H automation routine should be processed by JESOPER. If it is

called on another task it is routed back to JESOPER.v Each recovery action during the reset intervalv AOFRSD0H is only scheduled after the reset interval if no new recovery action

of the corresponding type SHORT or FULL has been taken during this time.v The pass counter for spool full recovery processing is reset by AOFRSD0H after

the reset interval, even if spool short recovery is still in progress.

Command = ACF ENTRY=JES2,TYPE=INITDRAIN,REQ=DISPSYSTEM = AOC1 AUTOMATION CONFIGURATION DISPLAY - ENTRY= JES2-------------------------------------------------------------------------------AUTOMATION CONFIGURATION DISPLAY - ENTRY= JES2TYPE IS INITDRAINLIN = ""YES""LOG = ""YES""OFF = ""NO""PRT = ""YES""RDR = ""YES""PUN = ""YES""END OF MULTI-LINE MESSAGE GROUP

Figure 43. DISPACF JES2 INITDRAIN Panel

AOFRSD0G


ExamplesThe following example shows a sample scenario for JES2 spool recoveryprocessing:

The following entries in the NetView automation table are used to issue theAOFRSD01 automation routine from the NetView automation table, when one ofthe expected messages arrives:IF MSGID(2) = ’HASP050’ & TEXT = .’TGS’.THENEXEC(CMD(’AOFRSD01’)ROUTE(ONE %AOFOPJESOPER%));IF MSGID(2) = ’HASP355’THENEXEC(CMD(’AOFRSD01’)ROUTE(ONE %AOFOPJESOPER%));

The SPOOLSHORT recovery is configured using the automation policy item JES2SPOOLSHORT as shown in Figure 44.

Because no smfids are defined, the own JES2 subsystem is responsible for JES2spool recovery processing. Editing JES2 SPOOLSHORT CMDS policy allows you toenter the pass recovery commands that are defined as shown in the response panelto command DISPACF JES2.

COMMANDS HELP------------------------------------------------------------------------------

JES2 SPOOLSHORT Specification Line 00000001Command ===> Scroll ===> PAGE

Entry Type : Application PolicyDB Name : DATABASE_NAMEEntry Name : JES2 Enterprise Name : YOUR_ENTERPRISE

Spool Shortage Retry Time . . . . . 00:02:00 (hh:mm:ss)Spool Shortage Buffer . . . . . . . 5 (0 to 50)Spool Shortage Reset Time . . . . . 00:15:00 (hh:mm:ss)

Enter priority of systems for spool recovery:Spool Shortage Recovery SMFID 1 . . SYS1Spool Shortage Recovery SMFID 2 . . SYS2Spool Shortage Recovery SMFID 3 . . SYS3Spool Shortage Recovery SMFID 4 . .Spool Shortage Recovery SMFID 5 . .Spool Shortage Recovery SMFID 6 . .Spool Shortage Recovery SMFID 7 . .Spool Shortage Recovery SMFID 8 . .Spool Shortage Recovery SMFID 9 . .

Figure 44. JES2 SPOOLSHORT Recovery Definition

AOFRSD0H


||||||||||||||||||||||||||||||

Assume that a JES2 spool shortage problem is reported by the following message:$HASP050 JES RESOURCE SHORTAGE OF TGS - 80% UTILIZATION REACHED

This issues the AOFRSD01 automation routine by the appropriate NetViewautomation table entry. AOFRSD01 initiates the JES2 SPOOLSHORT recoveryprocess and sets an every timer to call the pass processing routine by issuingAOFRSD09 JES2 SHORT every 5 minutes, as defined in the customization dialogfor SPOOLSHORT processing, see Figure 44 on page 217.

AOFRSD09 redetermines the actual spool usage, compares it with the definedTGWARN of 80% and calculates the target of recovery as difference of TGWARNand the buffer value resulting in a value of 75. If this value is exceeded by theactual spool usage, all recovery commands with the PASS1 selection option in theconfiguration file for the SPOOLSHORT recovery type are issued. After the retryinterval of 5 minutes, AOFRSD09 is reissued by the timer.

If AOFRSD09 now determines that the JES2 spool shortage problem has beenrelieved, it stops recovery processing and sets a timer to issue AOFRSD0H JES2 SHORTafter the reset interval of 15 minutes.

If none of the expected JES2 messages arrives by the end of the reset interval, theAOFRSD0H automation routine resets the pass count to 1 so that the nextSPOOLSHORT recovery process issues recovery commands beginning again atPASS1 selection option.

HASP099

RestrictionsShutdown processing of the JES2 message HASP099 is only done if:v Shutdown automation for JES2 is onv JES2 is in the process of being shut down

UsageThe ISSUEACT command responds to message:HASP099 ALL AVAILABLE FUNCTIONS COMPLETE

This indicates that all JES2 job processors have become dormant, and no JES2 RJElines are active.

Command = ACF ENTRY=JES2,TYPE=*,REQ=DISPSYSTEM = KEY3 AUTOMATION CONFIGURATION DISPLAY - ENTRY= JES2-------------------------------------------------------------------------------TYPE IS SPOOLSHORTCMD = (PASS1,,’MVS $PQ,Q=N,A=3’)CMD = (PASS1,,’MVS $OQ,Q=N,A=3,CANCEL’)CMD = (PASS1,,’MVS $PQ,Q=V,A=3’)CMD = (PASS1,,’MVS $OQ,Q=V,A=3,CANCEL’)CMD = (PASS2,,’MVS $PQ,ALL,A=4’)CMD = (PASS2,,’MVS $OQ,ALL,A=4,CANCEL’)CMD = (PASS3,,’MVS $PQ,ALL,A=3’)CMD = (PASS3,,’MVS $OQ,ALL,A=3,CANCEL’)CMD = (PASS4,,’AORRSPLS RANGE=JOB1-5000,NAME=T*’)CMD = (PASS4,,’AORRSPLS RANGE=JOB5000-10000,NAME=T*’)CMD = (PASS4,,’AORRSPLS RANGE=JOB10000-15000,NAME=T*’)CMD = (PASS4,,’AORRSPLS RANGE=JOB15000-20000,NAME=T*’)

Figure 45. DISPACF Command Response Panel

AOFRSD0H


INGRMJSP

PurposeYou use the INGRMJSP automation routine to monitor JES2 spool file usage. Itqueries the spool usage to obtain the current spool usage and the warning level. Ifnecessary it calls the INGRCJSP automation routine for JES2 spool recoveryprocessing.

The INGRMJSP command also updates the SPOOL entry in the status displayfacility (SDF) every time it is called.

Syntax

►► INGRMJSP ►◄

RestrictionsMonitoring by INGRMJSP is only done if it has been defined as the monitorcommand for an appropriate monitor resource in the customization dialog.

UsageThe INGRMJSP monitoring routine queries the spool usage by issuing the DSPOOLDEF,TGSPACE command to obtain the current spool usage and the warninglevel as set up by the JES2 system programmer:v If the spool file is full, INGRMJSP sets the health status to CRITICAL and calls

INGRCJSP.v If the spool usage is above the warning level, INGRMJSP sets the health status

to WARNING and calls INGRCJSP.

Depending on the spool full percentage and the warning level, one of thefollowing return codes is set:

Return code Meaning

1 A severe error occurred:

v The monitor does not have a job name

v The monitored object is not SPOOL or associated with JES2

v The specified job name does not refer to a JES2 resource

v No command prefix for JES2 was found

2 Monitoring command failed:

v The D SPOOLDEF command failed

3 OK: Spool usage is below the warning level

4 WARNING: Spool usage is above the warning level

6 CRITICAL: The spool file is full

ExampleTo create a spool usage monitor in the customization dialog you must define thefollowing items:

INGRMJSP


1. A monitor resource (MTR) with INGRMJSP as the monitoring command. Forexample, if you create a monitor resource called JES2SPOOL with the shortdescription JES2 Spool Monitor, specify the following information in theMONITOR INFO policy item:

MONITOR INFO item Entry

Monitored Object SPOOL

Monitored Jobname JES2

Activate Command

Deactivate Command

Monitor Command INGRMJSP

Monitoring interval 00:15

Captured MessagesLimit

20

Desired Available

Inform List SDF

Owner

Info Link

2. The following relationships to the JES2 application using the RELATIONSHIPSpolicy item:

Relationship type Supporting Resource Condition

HasParent JES2/APL/=

ForceDown JES2/APL/= WhenObservedDown

The monitor has a HasParent relationship to the corresponding JES2 resourcebecause it only makes sense to monitor the spool usage when JES2 is active.

3. The following recovery actions in the HEALTHSTATE policy item:

State Command

WARNING INGRCJSP

CRITICAL INGRCJSP

INGRCJSP (AOFRSD01)

PurposeYou can use the INGRCJSP automation routine for JES2 spool recovery processing.It responds to JES2 spool shortage messages by initiating the recovery process forJES2 spool shortage. It responds to JES2 spool full messages by initiating therecovery process for JES2 spool full to downgrade the problem of excessive spoolusage.

The INGRCJSP routine does the following:v Makes linear and first order predictions of spool usage, based on actual and

historical values.v Posts the spool status to the status display facility (SDF).

INGRMJSP


v Determines the target of recovery processing as the difference between the actualwarning threshold for track groups and the buffer value from the configurationfile. The spool shortage condition is considered as relieved if the recoveryprocess achieves this target.

v Initiates pass processing to execute the recovery commands of the configurationfile, as defined with the JES2 SPOOLSHORT or JES2 SPOOLFULL policy item.The pass processing itself is done by the AOFRSD09 automation routine, whichis issued every retry interval. The retry interval is taken from the configurationfile.

You define recovery commands and configuration parameters for JES2 recoveryprocessing, such as buffer value and retry interval, using automation policy itemJES2 SPOOLSHORT for spool shortage recovery processing and JES2 SPOOLFULLfor spool full recovery processing.

For further information about the JES2 SPOOLSHORT and JES2 SPOOLFULLautomation policy items see IBM Tivoli System Automation for z/OS DefiningAutomation Policy.

INGRCJSP should be called from the NetView automation table.

Syntax

►► INGRCJSP ►◄

Restrictionsv Processing in INGRCJSP is only done if it is called from NetView automation

table by JES2 messages HASP050 or HASP355.v Message HASP355 is only processed if it reports a shortage of track groups (TG).

UsageThe INGRCJSP automation routine is intended to respond to the followingmessages:HASP050 JES2 RESOURCE SHORTAGE OF TGs - nnn% UTILIZATION REACHED

HASP355 SPOOL VOLUMES ARE FULL

HASP050 indicates that JES2 has a shortage of track groups and the current spoolutilization exceeds the current TGWARN value on this JES. TGNWARN is definedin the SPOOLDEF statement in the JES initialization member and can be changeddynamically.

HASP355 indicates that a request for JES2 direct access spool space cannot beprocessed because all available space has been allocated to JES2 functions or nospool volumes are available. Therefore the recovery targets in this case are basedon a figure of 100% spool utilization.

You should code TGWARN in the SPOOLDEF statement in the JES initializationmember so that SPOOLSHORT recovery is initiated before a SPOOLFULLcondition is reached. If you do not do this the recovery process may becomeunpredictable.

INGRCJSP (AOFRSD01)


When resetting after a SPOOLFULL condition, the problem is downgraded to aSPOOLSHORT condition. SA z/OS expects the SPOOLSHORT recovery that waspreviously running to activate and try to downgrade the problem to an OK.Without the prior SPOOLSHORT recovery, the spool status remains inSPOOLSHORT after a successful SPOOLFULL recovery.

The NetView automation table entries for JES2 messages must respect the onecharacter prefix in front of the message identifier of JES2 messages that identifiesthe issuing JES.

The spool status is posted to SDF under the SPOOL generic, with the name of thesubsystem as its specific name. To have these displayed on an SDF panel, you needstatus fields for xxxx.SPOOL, elements 1 through n, where n is the number ofdifferent subsystems that use the spool.

INGRTAPE

PurposeINGRTAPE maintains tape status details under SDF and NMC. When SA z/OSdetects an outstanding tape mount request then it feeds the related message intoSDF and/or NMC. If the request is not satisfied before the warning interval hasexpired, the status will change to warning.

If the tape mount request is still not satisfied after the alert delay, the status willchange to alert.

The tape mount request is deleted from SDF and/or NMC dynamically when therelated tape is mounted or the requesting job is canceled.

The routine INGRTAPE automation routine is used to visualize the pending tapemount requests within SDF. Its behaviour is based on the definitions in the 'TapeAttendance' policy entry. For information about activating and customizing Tapeattendance, refer to IBM Tivoli System Automation for z/OS Defining AutomationPolicy.

Syntax

►► INGRTAPE ►◄

UsageAutomation routine INGRTAPE is intended to respond to the following messages:IEC501E, IEC501A, IEC502E, IEC503I, IEC507D, IEC509A, IEC510D, IEC512I,IEC513D, IEC514D, IEC701D, IEC702I, IEC703I, IEC704A, IEC706I, IEC707I,IEC708I, IEC708D, IEC709I, IEC710I, IEC711I, IEC712I, IEC713I, IEC714I, IEC715I,IEF233A, IEF233D,IEF234E, IEF455D, IAT5210, TMS001, TMS002, TMS0012

RestrictionsThe monitoring of tape mounts is only enabled when activated via theCustomization Dialogs.

INGRCJSP (AOFRSD01)


Actions are only taken in INGRTAPE if the recovery automation flag is on for themessage in question. The flag check is performed against minor resourceMVSESA.messageID.

INGRX711

Purpose

Automation routine INGRX711 can reformat the primary and alternate LOGRCDSs with an increased DSEXTENT parameter whenever the system reports adirectory shortage.

Syntax

►► INGRX711 ►◄

Usage

Automation routine INGRX711 is intended to respond to messages IXG261E andIXG257I.

Restriction

Automation must be allowed for the minor resource names: LOGGER and CDS.

INGRX740

PurposeYou can use the INGRX740 automation routine to respond to some syslog relatedsystem messages by issuing defined recovery actions from the automation controlfile to restart the syslog or to assign the syslog as a hardcopy medium.

INGRX740 keeps track of the incoming IEE037D syslog inactive message andcompares its occurrence with predefined thresholds for the MVS component minorresource, LOG. As long as the critical threshold level is not exceeded, a recoveryaction related to a previously received system message is issued.

If one of the messages IEE043I, IEE533E or IEE769E is received prior to theIEE037D message that is currently being processed, the commands that have beendefined for IEE043I, IEE533E or IEE769E in the MVSESA/msgid entry/type-pair ofthe configuration file are issued. If none of these messages has been received priorto the IEE037D message that is currently being processed, the command MVSWRITELOG START is issued.

The recovery routine INGRX740 also responds to an incoming IEE041I message ifthis indicates that the SYSLOG data set is available for use as a hardcopy log.Commands are issued in response to message IEE041I that are defined in theMVSESA/IEE041I entry/type-pair of the configuration file. An appropriatecommand in this case would be MVS VARY SYSLOG,HARDCPY to have theSYSLOG receive the hardcopy log.

INGRX740 should be called from the NetView automation table.

INGRTAPE


Syntax

►► INGRX740 ►◄

Restrictions and LimitationsProcessing in routine INGRX740 is only done if the following conditions are met:v The recovery automation flag for LOG is on.v The routine is running on an automation task.v The routine is called from NetView automation table by one of the expected

messages– IEE037D– IEE041I– IEE533E– IEE769E– IEE043I

Actions in response to message IEE037D are only taken in INGRX740, if the JobEntry Subsystem is up and running.

UsageAutomation routine INGRX740 responds to the following messages:IEE037D LOG NOT ACTIVEIEE041I THE SYSTEM LOG IS NOW ACTIVE[-MAY BE VARIED AS HARDCOPY LOG]IEE043I A SYSTEM LOG DATA SET HAS BEEN QUEUED TO SYSOUT CLASS classIEE533E SYSTEM LOG INITIALIZATION HAS FAILEDIEE769E SYSTEM ERROR IN SYSTEM LOG

ExampleThis example shows a sample scenario for system log failure recovery.

The following entry in the NetView automation table is provided by SA z/OS toissue INGRX740 in response to incoming messages IEE043I and IEE037D:IF MSGID = ’IEE037D’ THENEXEC(CMD(’INGRX740’)ROUTE(ONE %AOFOPRECOPER%));IF MSGID = ’IEE043I’ THENEXEC(CMD(’INGRX740’)ROUTE(ONE %AOFOPRECOPER%));

Assume that the following threshold levels are defined in the automation policyfor MVS component minor resource, LOG.

INGRX740


Assume that a command is defined for message IEE043I in the automation policyitem MESSAGES/USER DATA of MVS components, as shown in the followingfigure.

Assume that the following messages arrive the first time for one day, while the JobEntry Subsystem is up and running and the recovery automation flag for the MVScomponent minor resource LOG has not been switched off:IEE043I A SYSTEM LOG DATA SET HAS BEEN QUEUED TO SYSOUT CLASS 1IEE037D LOG NOT ACTIVE

Because IEE043I has been received prior to message IEE037D and the criticalthreshold that has been defined for message IEE037D has not been exceeded, thecommand that has been defined for message IEE043I is issued in response tomessage IEE037D.

INGRYSHU

Purpose

You can use the INGRYSHU routine to issue commands when, and only when, thesystem is involved in a system shutdown, controlled by either GDPS or triggeredwith the INGREQ ALL command.

COMMANDS HELP------------------------------------------------------------------------------



Resource : MVSESA.LOG




Figure 46. Threshold Definitions for MVS Component LOG

COMMANDS HELP------------------------------------------------------------------------------

CMD Processing Row 1 to 4 of 20Command ===> SCROLL===> PAGE

Entry Name : MVS_COMPONENTS Message ID : IEE043I

Enter commands to be executed when resource issues the selected message.or define this message as status message.

Status . . . (’?’ for selection list)

Pass/Selection Automated Function/’*’Command Text

MVS WRITELOG START

Figure 47. MESSAGES/USER DATA Policy Item for Entry/Type-Pair MVSESA/LOG

INGRX740


|

|

|||

Syntax

►► INGRYSHU command ►◄

Parameterscommand

The command that should be issued when a system shutdown is inprogress.

Restrictions and Limitations

Before the command is issued, INGRYSHU will verify that a system shutdown isin progress.

Usage

INGRYSHU can be invoked from the SHUTDOWN policy but is not restricted tothat usage.

Example

This example shows the SHUTINIT policy entry that could be used during JES3shutdown and the command should only be issued when a system shutdown is inprogress:INGRYSHU MVS F BPXOINIT,SHUTDOWN=FORKS

This example shows the SHUTFINAL policy entry that could be used during JES3shutdown and the command should only be issued when a system shutdown is inprogress:INGRYSHU MVS F BPXOINIT,SHUTDOWN=FILESYS

INGRYSHU


|

||||||||||

|

|||

|

||

|

||

|

|||

|

|||

|

|

Chapter 19. Automated System Resource Discovery

The discovery utility can be run against one or multiple customer systems,extracting all automation relevant data from the currently active address spaces.

That will let SA z/OS make a simple link between the address spaces details (jobname, command prefix, start command, and so forth) and a class definition thatprovides messages, dependencies and appropriate automation policy. It runsthrough the control blocks, pulling out not only job names, but also procedure andstep names. It pulls out the start commands, and the PGM name from the JCL. Itcan search through CICS and IMS implementations to pull out lists of dependentregions. All that data is then imported into the customer's Policy DB which willcreate a basic automation policy for them.

This policy then requires manual steps to create a ready to run automation policy.The policy will contain:v Sysplex Group(s)v Systemsv A System APG w/o Automation name where all applications found on that

system are linked tov APL Classesv APL Instances, linked to classesv Basic dependency structures

Note, there is no intention to discover any service period (schedule) related data.Furthermore, only applications that are running when the discovery utility is runwould be included in the SA z/OS policy.

In a subsequent step, rerunning the discovery utility and re-importing it willupdate the policy DB to add any newly discovered APLs.

The discovery utility can be used when:v installing and setting up SA z/OS,v customers who need basic automation want a simplified way to maintain their

policies.

Note: Automated System Resource Discovery is designed to support you whensetting up a new policy. The generated policy requires manual steps to adapt thepolicy to your installation. You therefore need to be familiar with the SA z/OSCustomization Dialogs.

DisclaimerThe Autodiscovery function provides a basic automation policy which must befinalized. It is the responsibility of the user to finalize the policy in a way that doesnot cause unwanted effects in the user's installation.


Components OverviewThere are three major process elements to this design. They are the discoveryengine, the preloader and the file update facility of the SA z/OS CustomizationDialog, here called 'importer'. In this overview you will read about the purpose ofthese processes and the data they require and generate.

DISCOVERY ENGINE

The DISCOVERY engine runs on each target system. It non-disruptivelyextracts information from the system's current workload and writes it to asystem specific SNAPSHOT file.

SNAPSHOT FILEThe SNAPSHOT file is a regular text file and can be browsed with the ISPFbrowser. It should not be edited. It mainly contains resources such asaddress spaces which were active at that point in time when the discovery

UserBestPractice

Preloader

SnapshotFile

Report

TargetPDB

Knowledge Base

SA z/OSBestPractice

Flat File

MapperFiles

User

SA z/OS

Target System

DiscoveryEngine

Customization Dialog System

SnapshotFile

Importer

SA z/OSCustomizationDialog

XMIT

Figure 48. Automated Discovery Overview

Components Overview


engine was running. The snapshot file is then sent over to the centralsystem where the customer is running the SA z/OS CustomizationDialogs.

PRELOADER

The PRELOADER reads the data from the SNAPSHOT file, applies therules encoded in the MAPPER FILES to the data, extracts the relevantpolicy from the KNOWLEDGE BASE and then determines a delta updatethat needs to be made to the PDB to add the newly discovered data. Itwrites out a set of instructions for the IMPORTER in a FLAT FILE.

KNOWLEDGE BASE

The KNOWLEDGE BASE represents two automation policies, a bestpractice policy provided by SA z/OS and an installation specificautomation policy. The SA z/OS provided policy already contains avariety of different applications that are expected to be encountered on acustomer's system. This policy cannot be changed by users. Theinstallation's automation policy is a PDS that must be created by theinstallation using the Customization Dialog. It is intended to be populatedwith automation policy items to be used as a model for entities that youhave identified in your installation's MAPPER FILES.

MAPPER FILES

The MAPPER FILES primarily describe how the PRELOADER has to mapa resource listed in the SNAPSHOT file to a policy entry contained in theKNOWLEDGE BASE. Besides those mapper files, which are predefined bySA z/OS , there may be installation defined mapper files. The preloaderwill first try to map a discovered resource using the installation's mapperfile. If no mapping was found, the preloader will continue using theSA z/OS provided mapper file. The MAPPER FILES are regular text filesand can be processed with a normal text editor.

PDB

The PDB - also referred to as 'Target PDB' - is an SA z/OS Policy Databasethat the user has created to hold the definitions of the discovered data. Theuser may have manually updated it to add or customize specific policies.

FLAT FILE

The FLAT FILE is read in by the IMPORTER, and the instructionscontained within it are then executed to edit the PDB that it is beingimported into. This will create the policy elements that represent thediscovered data.

FLAT FILES are regular text files and can be processed with a normal texteditor. They should be edited with caution however as they both need tofollow the syntax rules for the SA Customization Dialogs file update utilityand the introduction of even a small error or incompatibility will preventthe whole file from loading.

IMPORTER

The IMPORTER is just a short term for the file update facility of theSA z/OS Customization Dialog. It reads a FLAT FILE and imports thedata into the Target PDB.

Components Overview

Chapter 19. Automated System Resource Discovery 229

Overview of Using the Automated System Resource DiscoveryProcess

The Automated System Resource Discovery process can be considered as twoseparate steps: Gathering data and feeding the discovered data into a policy.

Step 1: Scan each of the systems to be modelled with a discovery tool - the'Discovery Engine'. For that purpose, you will need to install this tool on eachsystem to be modelled, unless there is a full SA z/OS installation on it. When thediscovery engine is run, it produces a SNAPSHOT data file containing informationabout the system resources. This snapshot file then needs to be sent to the systemwhere you will run the SA z/OS customization dialogs.

Step 2: During this step the snapshot files are consolidated and accumulated into aPolicy Database. The consolidation process involves:

Step2a: running each discovered data file through a PRELOADER which willapply some identification heuristics to the discovered data and construct a deltaupdate to the PDB in order to:Step2b: add in the newly discovered applications through the IMPORTER.

At the start of Step 2 you need to identify the PDB that you are going to add thedata to – it can be a newly allocated one or an existing one. You then run eachdata file through the preloader (Step 2a) and import it into the PDB (Step 2b)before you process the next data file.

During the preloader's analysis phase the discovery data is run through a set ofheuristic rules, as defined in MAPPER FILES, to classify it. Once it has beenclassified, its classification is utilized to select the automation policy that will beused as a model for it during the construction phase.

You may specify your own policy selection rules to supplement those supplied andmaintained by System Automation. System Automation provides 'best practice'automation policy models for many applications – and provides the ability for youto supplement or replace them with your own policy models. The output of all ofthis processing is a flat file containing PDB update statements that can be appliedto the PDB through the Customization Dialog's Data Management function.

You have an opportunity to inspect the flat file and the reports produced by thepreloader before you import it into the PDB, so you can return to the analysis andconstruction process to refine the identification and selection rules and then rerunthe preloader, if it is necessary to do so. Once you have imported the flat file intoyour PDB, you can move onto running the next data file through the preloader.

Step 1: Using The Discovery EngineThe 'DISCOVERY Engine' scans each of the systems to be modelled and stores itsoutput in a system specific snapshot data file.

Overview of Using the Automated System Resource Discovery Process


Extraction of the Discovery EngineIf the systems you wish to automatically model do not have a full installation ofSA z/OS upon them, you will need to submit a supplied sample job to extract thediscovery engine from a SA z/OS Installation.

The sample jobs are located in the SINGSAMP data set.1. Job INGEDEX0 should be run on a system where a full SA z/OS is installed. It

needs to be edited to specify the userid and system that the discovery engineshould be sent to prior to submission.

2. Job INGEDEX1 should be sent to the target system.3. Receive the transmitted data sets on the target system.4. Job INGEDEX1 should then be run on the target system. It will unpack the data

sets sent by INGEDEX0 after allocating the appropriate data sets to hold them.It needs editing to specify some details of the data sets prior to submission.

You then need to make the code available to all systems where it will be run andto secure it to prevent unauthorized execution.

Preparing the Discovery JobsThere is a single piece of JCL that needs to be submitted to run the discovery for asystem. It has two parameters – the system affinity setting in the JCL and the nameof the output data set.

Prior to running the discovery job there is some set up that is required:1. APF authorize the library the discovery code will be executed out of. For a full

SA installation this has probably already been done.2. Ensure that the user you will be executing the discovery process under has

appropriate authority – permission to enter authorized mode on the z/OS side

Target System

DiscoveryEngine

SnapshotFile

INGEDDSC

Figure 49. Discovery Engine Overview

Step 1: Using the Discovery Engine


|

|||

and superuser access on the USS side. These permissions are only neededduring the running of the discovery code and may be revoked afterwards. Notethat this has to be done for each system.

3. Produce the JCL to discover each system, with the system affinity and outputdata set properly set. You can use the INGEDDSC sample member to do this.You need to edit it to specify some of the details before you run it.

Running the Discovery JobsWhen you are prepared, submit the discovery jobs to:v step through all address space control blocks and return information for each

address space that is active at the time the discovery runs. If the address spacehosts an USS process, information about the USS process are returned as well.

v provide information about all XCF groups defined for the local systemv provide information about all ARM elements defined for the local systemv provide information about all locally defined system symbols.

This data is then written to a snapshot file.

It is better to submit all the jobs to the discovery systems within the same sysplexat around the same time, as it provides a more consistent view of activity acrossthe sysplex. Ideally, the job should be run at a point in time when most of theapplications are active. If your workload changes significantly during the day orweek, you can rerun the discovery process (once you have saved the output datafiles) to get a view of the systems other workload. These can then be integratedinto the PDB, although you will need to manually create and link the ServicePeriods to automatically switch between the workloads.

Step 2: Building the Automation PolicyThis step is performed on that system, where you run the SA z/OS CustomizationDialog to build the target PDB.

Step 2a: The PreloaderThe Preloader will analyze the data in a single snapshot and produce acorresponding model of those resources within a flat file suitable for the dialogs toimport.

Step 1: Using the Discovery Engine


Gathering of the Snapshot FilesYou need to have identified the system where you run your SA z/OSCustomization Dialogs as this is usually where you want to run the preloader totransform the snapshot data files into flat files you can import through theCustomization Dialog's data management function.

Then the snapshot data files from the system discovery jobs need to be transmittedto the system where you will run the preloader and stored there.

Note: If you choose to run the preloader on a different system to the one whereyou keep your PDBs and run the Customization Dialogs, then you will need tomake the PDB and KNOWLEDGE BASEs available to it and to bring the flat filesback to your Customization Dialog system for import.

UserBestPractice

Preloader

SnapshotFile

Report

TargetPDB

Knowledge Base

temporaryfiles

SA z/OSBestPractice

Flat File

MapperFiles

User

SA z/OS

INGEDPLD

Figure 50. Preloader Processes

Step 2: Building the Automation Policy


|

|||

Preparing the Preloader JobBefore you can run the preloader you need to do a few things:

Customize the preloader job. A sample JCL is shipped as member INGEDPLDlocated in the ING.SINGSAMP data set. You need to customize a copy ofINGEDPLD as described in its prolog. Here are the major steps:1. SA z/OS Knowledge Database PDB: Identify the HLQ of your SA z/OS

installation's target library. The HLQ is used to locate the SA z/OS KnowledgeBase PDB. This automation policy is generated based upon best practicepolicies supplied by SA z/OS. It contains the models for all of the resourcesidentified in the System Automation provided mapper files (and for a fewothers).

2. User's Knowledge Database PDB: Create or locate your installation'sKnowledge Base PDB. Even though you will not have edited your installation'smapping files yet if this is your first time through, you need to have aninstallation's Knowledge Base PDB, which initially might be empty. Later onyou might want to populate this knowledge base. This is recommended if youneed to deviate from the SA z/OS knowledge base or if you want to enhancethe knowledge base with your own entries.

3. Target PDB: Select the Target PDB. This is the PDB that you are going toupdate. It is recommended that this is either a newly allocated, empty PDB orone that has previously been occupied with automatically modelled data. If youuse a PDB containing hand modelled data or data from our hand modellingsamples, you are likely to get a lot of duplication, as the automaticallymodelled entries tend to be more complete, containing data that isn'tnecessarily specified on a hand made models.

4. Snapshot File: Locate the snapshot file which you want to process with thispreloader run. For each target system, you will have a snapshot file containingall discovered system resources and definitions. You need to invoke thepreloader for each snapshot file separately. Before processing another snapshotfile with the preloader, the generated flat file must be fed into the target PDBby the importer. This is required because the preloader needs to know thecurrent content of the target policy to avoid generating invalid flat files.

5. Mapper files: Locate the mapper files. They are members of the SINGIMAPdata set. There are some SA ones and some user ones. The users’ mapper filesin a U. When you want to modify them, you should allocate a new PDS whereyou copy and maintain them. User mapper files must exist even though theyinitially may not contain any mappings. You will find the mapper filesdocumented in Appendix F, “Autodiscovery Mapper Files and Report Formats,”on page 293.

6. Flat File: This is the output file of the preloader. It must exist before thepreloader job is submitted. After the flat file was generated and before anotherflat file will be generated, it is mandatory to import it into the target policy.

7. Report Data Set: When generating a flat file, the preloader documents itsprogress in various members of a report data set. The report is alwaysgenerated and the partitioned data set must exist before starting the preloaderjob. You will find the report members documented in Appendix F,“Autodiscovery Mapper Files and Report Formats,” on page 293.

8. The Preloader takes two parameters that affect its' output: Unless you havebeen specifically requested to do so by service, you normally omit the DEBUGparameter.v DISPLAY, which causes the contents of generated reports to also appear in

JOBLOG.



v DEBUG, which causes detailed debugging information to appear in theJOBLOG. Specifying DEBUG forces the DISPLAY option, even if you havenot specified it.

You may wish to create one copy of the preloader job per target system, that is foreach snapshot file. You are now ready to run the preloader.

Running the Preloader JobThe sequence for running the preloader is:1. Verify the preloader job to point to the correct input and output data sets for

the first target system's snapshot file.2. Submit the preloader job.3. Inspect the output from the preloader.

Remember: Before processing the next snapshot file with the preloader, thegenerated flat file must be imported into the target PDB (see Step 2b). Then thetarget PDB is prepared to serve as input to the preloader job for its next run. Thissequence helps to avoid conflicts when importing flat files described below.

Step 2b: The ImporterThis is the Data Management function of the SA z/OS Customization Dialog toupdate a policy database with data from a file.

TargetPDB

IMPORTER

SA z/OSCustomizationDialog

Flat File

Figure 51. Flat File Processing



You can either use the customization dialogs to select the target PDB and then usethe Data Management option to import the flat file into the target PDB, or you canuse a batch job to perform the import. You should do so only once you are happywith the contents of the flat file, as it is virtually impossible to unimport it unlessyou have saved a copy of the PDB prior to performing the import.

Sample job INGEBFLT is provided as a sample batch job to run the import. Oncethe import is complete you can run the preloader against another data file. Youhave to run it this way – preload/import – as the preloader uses the informationfrom the extract to prepare a delta update in the flatfile, that is, one that containsthings that are different. If you were to run multiple preloads against the sameextract file, you could end up with conflicts between the data in the flat files thatwould produce an invalid model when they were imported.

Extending Automated ModellingSA z/OS ships with a large number of presets for IBM and some 3rd partysoftware. There will, undoubtedly, be address spaces on your system that it haseither not identified or which it has identified incorrectly or incompletely.

Some of this will be because the applications are unique to your systems, some ofit will be because the allowed customization of the application is such that we donot have a suitable 'hook' to hard code into the files that we ship to reliablyidentify it.

Mapping FilesThe Automated Modelling Tool is extended by adding statements to a set of usermapping files that are processed by the preloader. System Automation (SA)versions of these files are also provided and should be used in conjunction withyour user files to take full advantage of the automated modelling capabilities.

The mapping files are listed below. The first name is the SA member, the second isa user member that you should edit your own policy into. Refer also to theappendix section “Mapper Files” on page 293.

INGSMAID / INGSMAIUThis is the Address Space Identification Mapping file. It contains rules foridentifying Address Spaces from the data discovered about the addressspace.

INGSMGRP / INGSMGRUThis is the XCF Group identification file. It contains rules for identifyingthe XCF groups that were found to have been established.

INGSMGMB / INGSMGMUThis is the XCF Group Member identification file. It contains rules foridentifying members of XCF groups based upon the type of the group andthe name that the member joined the group with.

INGSMUID / INGSMUIUThis is the USS Address Space Identification Mapping file. It contains rulesfor identifying processes that were found running under USS.

INGSMPLU / INGSMPLYThis is the Policy Mapping file. If contains rules for selecting whichidentified Address Spaces will be converted into SA policy, what modelpolicy, if any, will be used for them and which of their established fieldswill be over written.



INGSMVRS / INGSMVRUThis is the variable mapping file. It is used to define symbols – in additionto those found by the discovery process – that can be used in the fieldvalue formulas in the policy mapping file.

The supplied user files contain some sample values that will probably not work onyour system. You should remove them.

Finalizing the Target PolicyAfter the systems have been discovered and the target policy has been populated,manual updates are required in order to get an operational automation policy.

The steps below will lead you to an operational policy which supports starting,stopping and restarting resources in place where all applications are linked to acommon application group (APG) per system.

Creating and preparing a Reference PolicyThere are a few definitions to be made in the target policy, which best can beimported from another policy. That policy is intended to serve as a referencepolicy. If you do not have such a policy yet, you may create it now while selectingthose Best Practice policies, which contain samples for the discovered applications.

As an example: if z/OS base components and DB2 were discovered, the *BASEand *DB2 Best Practice policies should be selected. This new policy also serves asthe source, where missing policy elements can be copied from later on.

Importing from the Reference PolicySwitch to the Target PDB. Navigate to the 'Import entries from a Policy Database'panel and select the reference policy as 'Source Policy Database'.

Enter 'GRP' as the 'Entry Type'and 'YES' in the 'Import Linked Entries' field. Thencontinue with selecting the sysplex(es) and proceed to panel 'Selected Entry Namesfor Import'.

This panel lists all policy entries as contained in the reference policy. It allows youto add additional entries to your target policy. Because all entries are imported bydefault -if they do not already exist- into your target policy, you need to:1. Overtype the GRP and SYS entry names in the 'Entry Name' column with those

names used in the installation, so they match the Sysplex and System names inthe target policy. As a result you will find a 'Y' in column 'D'.

2. Remove each entry from the list, which should not be imported. In order to getan initial operational policy, remove all entries from the 'Selected Entry Namesfor Import' panel except those entries of type MVC, SDF, ADF, AOP, NFY, NTWand XDF.

Then run the import.

If you have more sysplexes and/or systems in your target policy, you need torepeat this import step for each of the installation's sysplexes and systems.

Avoiding multiple entries for the same applicationIn some situations you may be able to reduce the number of policies required byspecifying values utilizing AOCCLONE variables.

Extending Automated Modelling


For example, if an application has an Application Restart Management (ARM)Element name like ABC@DEF and ABC@HIJ then you could choose to use anAOCCLONE value as the system dependent part. In the variable mapping file, adda rule for each system that specifies the correct system dependent value for thatsystem:PLEX1 SYS1 AOCCLONEA DEFPLEX1 SYS2 AOCCLONEA HIJ

Then in the user policy mapping file, specify that the ARM Element name for theaddress space should be modelled as ABC@&AOCCLONEA. You may need tocopy the mapping rule from the SA supplied mapping file.

This should result in the generation of an APL that has an ARM Element name ofABC@&AOCCLONEA. And which can be used on both the SYS1 and the SYS2systems. The AOCCLONEA values would also be set on those two systems.

In situations where the values cannot be specified in the APL policy usingAOCCLONE values, your options are more limited – clean up your system toreduce the variability or live with multiple application instances. In the latter case,you may want to create a class to hold the common policy and just leave theinstances holding the discovered data values. This will give you a single place tochange the automation policy for the APLs.

Changing the Name of an APL.You can use the dialogs RENAME function to rename APLs that have been createdby the automated modelling tool. Provided you do not change any of thediscovered data values the preloader will recognize the renamed APL if it is runagain and will not create a new APL.

This works because the preloader creates a signature for each APL, based upon thediscovered data values – Jobname, ARM Element name, Procedure Name, and soforth, and uses this to match discovered APLs to pre-existing APLs.

If you want to make the change on the preloader, find or copy the policy rule for itinto your user policy mapping file and change its entry name to reflect the nameyou wish it to have.

Target Policy Ready for BuildYour target policy is now ready for running the build process.

You can enhance the capabilities of this policy as you proceed with yourautomation needs. Importing from the reference policy can be repeated for furtherentries like application groups, monitors and transient applications. Their names,relationships and links need to be adapted according to the reference policy.

Building the Configuration Control DataYou may either build the Configuration Control Data using the online or batchbuild process. During the build process, WARNINGs may be issued. Please reviewthe build report after the build process completed.

WARNINGs due to the following reasons are expected:v relationships to non-existent APGs,v HASMONITOR relationships to non-existent MTRs,

Finalizing the Target Policy


v missing subtype specifications and subsequent warnings related to a missingsubtype definition

v missing Automation Name for automatically generated APGs(LK_sysname_plexname).

The steps below will lead you to an operational policy which supports starting,stopping and restarting resources in place where all applications are linked to acommon application group (APG) per system.

TroubleshootingMy Application is Missing From the Flatfile

Check first that the APL has not already been imported. You need to openthe PDB up in the customization dialogs and look at the APLs to checkthis. It may have been renamed. If the APL is already in the PDB, it willnot be included in the Flatfile.

Next you need to check the EXCLUDE report to see if the address space ismentioned in there. If it is, you need to look at the Policy Mapping rulethat it hit. If it is an exclusion rule, you need to add a new policy mappingrule to link the address space to the correct APL model for it.

For the APL import to work correctly, you need three things – correctidentification of the address space, model automation policy in one of theKB data sets (either an unmodified SA sample or an entry from your userKB) and a rule in the policy mapping files to tie the two together.

Multiple Entries are Generated for the same ApplicationSee the above section “Avoiding multiple entries for the same application”on page 237.

Building the Configuration Control Data


Troubleshooting


Appendix A. Global Variables

You must ensure that the names of any global variables you create do not clashwith SA z/OS external or internal global variable names. You should check thefollowing tables before creating any global variables of your own.

Read-Only VariablesThe values of these variables are used by SA z/OS internally. They can be readbut must not be changed by user REXX routines.

Table 24. Externalized Common Global Variables

Variable Name Description Reference

AOF.clist.0DEBUG Contains either a Y or blank. If it contains Yan intermediate level of debug that issupported by SA z/OS automationprocedures is turned on.

AOF.clist.0TRACE Contains a REXX trace setting to be used bythe automation procedure clist.

AOFAOCCLONEx Where x either does not exist(AOFAOCCLONE) or is a value from 1through 9 or A through Z. TheAOFAOCCLONEx global variables containthe values specified for the &AOCCLONEx.variables for this system.

See the description of theSystem policy object in IBMTivoli System Automation for z/OSDefining Automation Policy.

AOFBFP Contains the backup focal point.

AOFCFP Contains the domain ID of the current focalpoint.

AOFPFP Contains the primary focal point.

AOFCOMPL Contains YES if initialization is complete.

AOFDEBUG Contains a REXX trace setting to be usedglobally.

AOFINITIALSTARTTYP Contains the value 'IPL' or 'RECYCLE'depending on whether SA z/OS has beenstarted the first time after an IPL or after aNetView recycle.

AOF_PRODLVL Contains the release level of SA z/OS. Thevalues are:

SA z/OS 3.5SA z/OS,V3R5MO

SA z/OS 3.4SA z/OS,V3R4M0

SA z/OS 3.3SA z/OS,V3R3M0

AOFJESPREFX The command prefix for the primaryscheduling subsystem.

AOFSUBSYS The subsystem name of the primaryscheduling subsystem.


|||

||

||

||

|

Table 24. Externalized Common Global Variables (continued)

Variable Name Description Reference

AOFSYSNAME Contains the name of the system. See AOCUPDT in IBM TivoliSystem Automation for z/OSProgrammer's Reference.

AOFSYSTEM Contains the system type. Always set toMVSESA.

AOFSYSPLEX Contains the name of the physical sysplex. See the 'Sysplex' column on theINGAMS panel.

AOFSYSPLEXGROUP Contains the name of a GRP policy entrywith group type SYSPLEX as defined in thecustomization dialog.

The GROUP INFO panel of thecustomization dialog. See the'SAplex' column on theINGAMS panel.

Read/Write VariablesTable 25 lists the common global variables that can be user-defined. You can setthem in your startup exit to change the way that SA z/OS behaves. Thesevariables should be set only once for an SA z/OS system. You can enable ordisable advanced automation options (AAOs) by changing the settings of theglobal variables in your CNMSTGEN stylesheet. For example:******************************************************* System Automation AAO CGlobals******************************************************COMMON.AOFCNMASK = 290C0D0E0F101518COMMON.INGREQ_ORIGINATOR = 1COMMON.AOFRESTARTALWAYS = 0COMMON.AOFUPDRODM = NOCOMMON.AOFUPDAM = NOCOMMON.AOFSMARTMAT = 0

Table 25. Global Variables to Enable Advanced Automation (CGLOBALS)

Variable Value Effect

AOF_AAO_AAREQUEST_MAX_WAIT

1 to 999 seconds Defines the maximum wait time in seconds. Is usedwhen the MAXWAIT parameter is not specified for theworkload. See MAXWAIT parameter for valid range.

AOF_AAO_AOCUPDT_LOGGING YES | NO Specifies whether or not trace messages should bewritten to the NetView log in case AOCUPDT isinvoked without its LOG parameter specified. Thedefault is YES.

FORCE_YES |FORCE_NO

Specifies whether or not trace messages should bewritten to the NetView log overruling the valuespecified with the LOG parameter in the AOCUPDTinvocation.

AOF_AAO_GWPING_RETRY 1-99 The gateway monitor can issue the PING commandprior to attempting a potentially time consumingRMTCMD for a remote target. With this AAO you canspecify the number of retry attempts when the PINGcommand was not successful.

When the AAO is not set or when the value specified isinvalid then the PING command will not be issued atall.

Global Variables


||||

||||

||||

||||

||||||

||||||

Table 25. Global Variables to Enable Advanced Automation (CGLOBALS) (continued)


AOF_AAO_HW_VALIDATION(The option applies to SA z/OS only).

YES This option allows hardware validation to be performed.For details, see the chapter "How to enable SysplexAutomation" and the section "EnablingHardware-related Automation" specifically. YES is thedefault.

NO This option inhibits the hardware validation to continue.Functions depending on the validation may not workproperly. The affected validation routines terminate withreturn code 2.

AOF_AAO_INJECT_NOFORCE_REQ Any value SA z/OS does not inject a STOP vote with PriorityForce for source *RECYCLE when processing anINGREQ REQ=STOP RESTART=YES request. Insteadthe regular stop request is passed to the automationmanager and removed automatically when the resourceis down. This also removes any previous request for theresource that was made by the same source.

AOF_AAO_IPL_COMPLETE_MSG COND Specifies whether only message ING313I is given if allresources reached the AVAILABLE status within theexpected time limit and ING314I if this is not the case.

ALWAYS Specifies that if ING314I has been given and theresources become AVAILABLE later (at any point intime) ING315I will be issued additionally. Thesubscription remains in effect until the resources reachthe AVAILABLE status.

AOF_AAO_ISQ_KALIST ip_addr |ip_addr/i

Specifies either the dotted decimal address or thehostname of a SE or a HMC.

/ - separator

i - idle time in minutes

Multiple definitions can be made, separated by at leastone blank. Note, that each ip_addr must match with theprocessor address definitions from the PDB.

AOF_AAO_MSG_EHKVAR YES This indicates that when calling commands, the tokensof the triggering message are to be stored in variablesEHKVAR0 through EHKVAR9 and EHKVART, if notspecified in parameter EHKVAR.

YES is the default.

NO This indicates that the tokens of the triggering messageare not to be stored in EHKVAR variables, if notspecified in parameter EHKVAR.

AOF_AAO_MVSTAPEMON >0 Set this value to represent the number of iterations forINGRTAPE to continue monitoring using MVScommands after the LATE alert has been reached. Anon-zero entry also indicates using MVS commands forall tape mount monitoring prior to the LATE alert.

0 INGRTAPE relies on the receipt of the DOMMEDmessage to satisfy any outstanding alerts.

AOF_AAO_OMVS_SHUTDOWN NOWAIT This causes the wait for a complete termination ofOMVS to be skipped.

Global Variables

Appendix A. Global Variables 243

||||||||

|||||

|||||

||||||



AOF_AAO_RDS_TSO_DSN User-defined Must be set in order to use the RDS table editor underTSO. Specify a name of the RDS working data set. If thisvalue is changed then NetView requires a recycle for itto take effect.

AOF_AAO_RDS_TSO_RACFHLQ User-defined Must be set in order to use RACF protection of the RDStable editor under TSO. Specify high level qualifierwhich could be up to 26 bytes long and may containdots. If this value is changed then NetView requires arecycle for it to take effect.

AOF_AAO_RETENTIONPERIOD 0 to 1440 Defines how long (in minutes) SA z/OS should keepthe CGLOBALS that are used to keep track of commandrequests that are received from TWS. The default is 60minutes.

AOF_AAO_SDF_MSGDEL_OPERID taskname Name of the task to be used by RMTCMD to deletecaptured exceptional messages from the SDF focal pointon the target system.

<OPER> Use the original operator ID from the focal point forRMTCMD to delete the messages on the target.

blank Use the Reporter ID task for RMTCMD to delete criticalmessages on the target. This is the default.

AOF_AAO_SDFBFP_ROOT.* User-defined Defines the static root names of the backup focal point.For more details, refer to "Restrictions and Limitations"of the command SDFTREE in SA z/OS Programmer'sReference.

When used, the variable AOF_AAO_SDFBFP_ROOT.0must exist and defines the number of adjacent globals.

The value of an adjacent global can be the name of asingle root name or a list of root names separated by ablank character.

AOF_AAO_SDFROOT_LISTAOF_AAO_SDFROOT_LISTn(where n is 1 to 4)

User-defined (Deprecated, use AOF_AAO_SDFROOT.*)

Defines the value of the &SDFROOT variable that isused as the root name for the sample SDF panels thatare provided with SA z/OS.

The value can be the name of a single system or a list ofsystem names separated by a blank character. A list canbe used at the SDF focal point to have SA z/OSgenerate the necessary panel definitions for all systemsin the list.

Use the AOF_AAO_SDFROOT_LISTn variables to honorthe maximum length of common global variables withinthe NetView environment.

The system name can be appended by one or moremember names separated by slashes, the first namerefers to the panel definitions, and the second namerefers to the tree definitions, for example SYS1SYS2/MYPNLS SYS3&SLASH./MYTREE SYS4/MYPNLS/MYTREE.Refer to "Status Component Panel Definition" in IBMTivoli System Automation for z/OS Programmer's Referencefor the interpretation of these definitions.

Global Variables


||||||

|||||||

|||||

|||

|||

||

|||

|||||

|||

||||||||



AOF_AAO_SDFROOT.* User-defined Defines the value of the &SDFROOT variable that isused as the root name for the sample SDF panels thatare provided with SA z/OS.

When used, the variable AOF_AAO_SDFROOT.0 mustexist and defines the number of adjacent globals.

The value of an adjacent global can be the name of asingle system or a list of system names separated by ablank character. A list can be used at the SDF focal pointto have SA z/OS generate the necessary paneldefinitions for all systems in the list.

The system name can be appended by one or moremember names separated by slashes, the first namerefers to the panel definitions, and the second namerefers to the tree definitions, for example SYS1SYS2/MYPNLS SYS3&SLASH./MYTREE SYS4/MYPNLS/MYTREE.Refer to "Status Component Panel Definition" in IBMTivoli System Automation for z/OS Programmer's Referencefor the interpretation of these definitions. The defaultnames are AOFPNLS and AOFTREE.Note: NetView interprets two consecutive slashes as thebegin of a line comment. For this reason the sampleabove uses the symbol for the slash character followedby the slash character itself.

AOF_AAO_SDFTGT_COMP.* User-defined Defines particular component names to be forwarded tothe SDF focal point on RESYNC requests withoutdeleting and refreshing the whole tree of the root thatthe component belongs to. For more details, see"Restrictions and Limitations" of the SDFTREEcommand in SA z/OS Programmer's Reference.

When used, the variable AOF_AAO_SDFTGT_COMP.0defines the number of adjacent globals.

The trailing wild character asterisk (*) is supported onthe component name, the alternate component name,and the major component name, but not on the rootcomponent name. For example, the specification"ROOT.H*(IPL*)" matches all component namesbeginning with "H" and its alternate component namesbeginning with "IPL". In addition, system symbols aresupported as well.

AOF_AAO_SEC_PPIAUTH FAIL Used by 3rd party check for incoming commandreceived by the Command Receiver, Web Adapter, andE2E Adapter. If corresponding SAF profile or class is notdefined the execution of the incoming command isrejected.

not defined orany other value

Used by 3rd party check for incoming commandreceived by the Command Receiver, Web Adapter, andE2E Adapter. If corresponding SAF profile or class is notdefined the execution of the incoming command ispassed through.

Global Variables


|||||

||

|||||

|||||||||||||

|||||||

|||||||



AOF_AAO_SDFCxxx.* User-defined Defines the value of the &SDFCxxx variable that is usedas the status component name for the sample SDF treesthat are provided with SA z/OS where xxx can be anysequence of characters that meets the NetViewrequirements for common globals.

When used, the variable AOF_AAO_SDFCxxx.0 mustexist and defines the number of adjacent globals.

The value of an adjacent global can be the name of astatus component or a list of status component namesseparated by a blank character. All adjacent globals arecombined to a list to have SA z/OS replicate thedesired tree definitions for all component names in thelist.

For more information refer to "AOCUPDT" and"AOFTREE" in IBM Tivoli System Automation for z/OSProgrammer's Reference for the interpretation of thesedefinitions.

AOF_AAO_SDFCsaplex.n Names the System Automation subplexes of the entireenterprise.

AOF_AAO_SHUTDOWN_STOPAPPL User-defined Specifies the name of the resource (name/type/system)that SA z/OS monitors during system shutdown toreport shutdown complete back to GDPS. It is intendedto be used by installations that have both, GDPS andnon-GDPS environments, and that want to use the samesystem shutdown mechanism where a particularresource (STOPAPPL) is to be monitored. Note that thisoption is only effective when the TOWER.SA.GDPS isnot set. For other GDPS environments, for example,PPRC, the stop application must be defined inGEOPLEX OPTIONS entry type within UET pairs.

AOF_AAO_SHUTSYS_OLD YES Indicates that SA z/OS should not redirect the INGREQALL REQ=STOP command to the GDPS STOPAPPLresource when the GDPS tower is active.

AOF_AAO_TRANRERUN YES This indicates that a transient job can be rerun withinthe lifecycle of a particular z/OS, if not specifiedotherwise in the automation policy for this job.

NO This indicates that a transient job is only run once in thelifecycle of a particular z/OS, if not specified otherwisein the automation policy for this job. NO is the defaultvalue.

AOF_AAO_TWS_CHK_CONDDEP YES Set this AAO to YES to exploit the new behavior thatallows you to save the return code passed back from thecompletion checking routine for conditional dependencyprocessing in case of successful operations.

AOF_AAO_TWS_CHK_OUTPUT_NETLOG

YES Set this AAO to YES to place the output of thecompletion checking routine in the netlog.

AOF_AAO_TWS_CMD_OUTPUT_NETLOG

YES|NO Set this AAO to YES to place the output of thecommand execution in the netlog.

AOF_AAO_TWS_ERRMSG This AAO can be used to inhibit the ERRMSGparameter. If set to NON BLANK, it erases the contentsof the ERRMSG parameter.

Global Variables


|||||||

||

||||||

||||

||||

|||||||||||||



AOF_AAO_TWS_MAX_WAIT_TIME Defines the installation default for the maximum waittime for the INGREQ and INGMOVE command. Thedefault is taken when no wait time is specified in thecompletion information parameter.

AOF_AAO_TWS_RESYSPLEX YES|NO This AAO can be used to allow the TWS specialresource name to use the SA z/OS Sysplex nameinstead of SYSPLEX to facilitate an enterprise widenaming convention.

Default: NO for SYSPLEX

AOF_AAO_VPCEINIT 0 SA z/OS does not invoke the GDPS initialization exit,VPCEINIT,

AOF_ASSIGN_JOBNAME 1 This indicates that SA z/OS exploits the NetView"ASSIGN BY JOBNAME" feature with a higher prioritythan the "ASSIGN BY MESSAGE ID" feature (prioritylevel 3).

This is the default setting.

0 SA z/OS exploits the NetView "ASSIGN BY JOBNAME"feature with a lower priority than the "ASSIGN BYMESSAGE ID" feature (priority level 4).

AOF_E2E_EAS_PPI User-defined PPI receiver ID of the event/automation service to beused to forward events to the end-to-end automationadapter.

AOF_E2E_EVT_RETRY 1 to n Specifies the number of retries, at intervals of onesecond, that are used to transfer events via PPITECROUTE to the message adapter of theevent/automation service. The events are thenforwarded to the end-to-end automation adapter.

AOF_E2E_EXREQ_NETLOG 1 The output to requests received from the end-to-endautomation adapter and issued by the primaryautomation agent, is logged to the NetView log.

0 The output to those requests is not logged to theNetView log.

0 is the default setting.

AOF_E2E_TKOVR_TIMEOUT hh:mm:ss If a hot restart of the automation manager takes longerthan the value specified in this variable, the end-to-endautomation manager is informed about the outage andhas to resynchronize with the first-level automation.

AOF_EMCS_AUTOTASK_ASSIGNMENT

1 SA z/OS assigns an autotask to extended MCS consoleswith a console status of MASTER or ACTIVE

0 SA z/OS does not assign an autotask to extended MCSconsoles with a console status of MASTER or ACTIVE

0 is the default.

Global Variables




AOF_EMCS_CN_ASSIGNMENT 1 SA z/OS obtains an extended MCS console with aunique name for operator station tasks (OSTs). If anMVS console was obtained for the OST previously, it isreleased.


0 SA z/OS does not obtain an extended MCS consolewith a unique name for OSTs and the commandAOCGETCN is disabled.

AOFACFINIT 1 This indicates that SA z/OS attempts to proceed withinitialization despite error messages during theprocessing of the automation control file.


0 SA z/OS stops the initialization process upon sucherrors.

AOFARMQUERYRETRYS User-definednumeric value

The number of times AOFPARMQ is called to query theARM status of an element after a status of UNKNOWNis returned. If the ARM status does not change toanother status before the number of retries is exhausted,SA z/OS continues processing and assume the elementis not ARM-enabled.

The default is 10.

AOFARMQUERYWAIT User-definednumeric value

The number of seconds to wait between retries asspecified in the AOFARMQUERYRETRYS value above.

The default is 15.

Global Variables




AOFCNMASK User-defined The characters that are used in determining uniqueconsole names can be tailored by updating the commonglobal variable AOFCNMASK. This global is used as ahex mask to extract characters from the following stringwhen generating unique console names with commandAOCGETCN:

left(opid(),8)||right(opid(),8),||left(aofsysname,4)||right(aofsysname,4),||left(applid(),8)||right(applid(),8),||’ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789$#@_!’,||aofcnmask_extended

Where

v opid() is a function that returns the OST task name

v aofsysname is a common global that stores the systemname

v applid() is a function that returns VTAM LU name

v aofcnmask_extended is an optional common globalthat can store an additional user-defined identifier(see AOFCNMASK_EXTENDED.)

The default for AOFCNMASK is 290C0D0E0F101718.X'29' selects character A in position 41, X'0C' throughX'10' selects the last five characters of the opid inpositions 12 to 16, X'17' and X'18' select the last twocharacters of the sysname in positions 23 and 24.

If AOFCNMASK is null, AOCGETCN attempts to obtaina unique extended MCS console after a 1 minuteinterval, followed by a two minute interval and so forthfor a maximum of 5 passes (15 minutes elapsed from theinitial invocation of the command).

Tailoring AOFCNMASK requires you to ensure that itconsists of an even number of characters, because everytwo characters represent one hexadecimal number.Otherwise the generation of unique console names doesnot work.

For example, with

AOFCNMASK: 2A01020304055455

X'2A' selects character B in position 42, X'01' throughX'05' selects the first five characters of the opid inpositions 1 to 5, X'54' and X'56' select the last twocharacters of the sysname in positions 84 and 85.

Global Variables




AOFCNMASK_EXTENDED User-defined Additional identifier for the generation of uniqueconsole names.

This variable can be set individually in the NetViewstylesheet, if an additional user-defined identifier isneeded for the generation of unique console names (forexample, if previous console names were not unique).

In this case AOFCNMASK_EXTENDED has to beassigned with a value and the common global variableAOFCNMASK has to be tailored to extract the neededcharacters from AOFCNMASK_EXTENDED (seeAOFCNMASK).

If an additional identifier is not needed for thegeneration of unique console names, this commonglobal does not have to be used and the console namesare generated in the usual way.

Maximum length is 172 characters.

AOFDEFAULT_TARGET User-defined Sets a default for the TARGET parameter for allcommands where this parameter is used.

AOFDESCA 0100001000001000

Descriptor code for action messages

AOFDESCD 0100001000001000

Descriptor code for decision messages

AOFDESCE 0010001000001000

Descriptor code for eventual action messages

AOFDESCI 0000011000001000

Descriptor code for informational messages

AOFDESCW 1000001000001000

Descriptor code for wait messages

AOFEXPLAIN_USER User-defined The EXPLAIN command accepts this variable to includehelp support for customer installation supplied terms. Itcan hold one or more pairs of term/help panelspecifications separated by a blank. If the specifiedstatus in the EXPLAIN command is not a validSA z/OS status, the command routine checks whether itis an installation defined term. If so, the associated helppanel is displayed.

AOFINITREPLY hh:mm:ss The initial reply AOF603D is issued and automaticallyresponded after hh:mm:ss.

00:02:00 (2 minutes) is the default setting.

0 The initial reply AOF603D is not issued and automationcontinues with the default start without asking theoperator.

AOF_INIT_MCSFLAG User-definedvalid value

This variable contains the MCSFLAG that is used forWTOs and WTORs that are issued by SA z/OS duringinitialization.

The default is '00000000'.

Global Variables




AOF_INIT_ROUTCDE User-definedvalid value

This variable contains the ROUTCDE (routing code) thatis used for WTOs and WTORs that are issued bySA z/OS during initialization.

The default is '01000000'.

AOF_INIT_SYSCONID User-definedvalid value

This variable contains the SYSCONID that is used forWTOs and WTORs that are issued by SA z/OS duringinitialization.

The default is blank.

AOFLOCALHOLD 0 INGNTFY and SA z/OS initialization executes theSETHOLD AUTO command on the notify operator.


1 SETHOLD must be manually invoked.

AOFMATLISTING 0 Setting this variable means that the NetView automationtable listing is not placed in the DSILIST data set atNetView automation table load time.

AOFOPCCMDMSG 0 OPCAMOD only produces messages that are generatedby INGOPC.


1 OPCAMOD produces EVJ011I, EVJ412I, EVJ420I, andEVJ423I messages.

AOFPAUSE 0 to 5 This is the number of seconds that SA z/OS allows forapplications that have shut down to be cleared by MVS,in addition to their termination delay. As theAOFPAUSE value is applied to all applications it shouldbe kept small. AOFPAUSE may be useful on a slowmachine, where allowing an extra second or two beforeSA z/OS checks if the application has been clearedcould avoid the need to use a termination delay timer.

No matter how AOFPAUSE is set, the application statusis not updated to AUTODOWN or CTLDOWN untilSA z/OS is sure that the application has been clearedfrom the system by MVS.


AOFRESTARTALWAYS 1 An application that has been shut down normally,outside the control of SA z/OS, withRESTARTOPT=ALWAYS, is restarted regardless ofwhether or not it has reached its critical error threshold.

0 An application that has been shut down normally,outside the control of SA z/OS, withRESTARTOPT=ALWAYS, is not restarted if it hasreached its critical error threshold.


AOFRMTCMDWAIT See NetViewRMTCMD

Contains the installation wait time when RMTCMD isused for communication.

60 seconds is the default setting for RMTCMD.

Global Variables




AOFRPCWAIT 0 to n This is the number of seconds that SA z/OS waits forcommand responses from other systems in the sysplex.


AOFSENDALERT Yes or No This defines whether NetView alert forwarding (YES) orthe command handler (NO) is used to forward data tothe focal point.

Yes is the default setting.

AOFSERXINT 1 The exit AOFEXINT is processed under the BASEOPERautomation operator under the initialization process.This is the default.

0 The exit AOFEXINT execution is serialized within theinitialization process.

AOFSHUTDELAY 0 to 59 This is the number of minutes that SA z/OS waits for atermination message before continuing the shutdownprocess. Any values outside this range are treated as 0.With a setting of 0, message AOF745E is not issued.


Global Variables




AOFSMARTMAT 0 The SA z/OS Agent is disabled from refreshing ATs andthe MRT. The user must supply an AT memberINGMSG02 which is included when SA z/OS initiallyloads INGMSG01.

1 The SA z/OS Agent is enabled to refresh ATs when anINGAMS REFRESH is issued. The AT fragment built bythe customization dialog is not loaded; the user mustsupply a member INGMSG02 which is used instead.

The ATs are loaded after a successful test load. This willallow the agent to notify the AM about a load problemof the AT. The agent may notify the AM of an AT loadfailure, thus stopping the configuration refresh. TheSA z/OS Agent will not load any MRT.

2 The SA z/OS Agent is enabled to load the AT that isgenerated by the customization dialog and to refreshATs when an INGAMS REFRESH is issued. The AT thatis built by the customization dialog is dynamicallyloaded into storage as the INGMSG02 fragment.

The ATs are loaded after a successful test load. Thisallows the agent to inform the AM about any loadproblem for the AT. The agent may notify the AM of anAT load failure, thus stopping the configuration refresh.

The SA z/OS Agent will not load any MRT.

This is the default value.

3 The SA z/OS automation agent is enabled to load theMRT that is generated by the customization dialog andto refresh the MRT when an INGAMS REFRESH isissued. The AT built by the customization dialog isdynamically loaded into storage as the INGMSG02fragment. The MRT built by the customization dialog isdynamically loaded as INGMRT01. The ATs and theMRT will be loaded after a successful test load. If anerror is detected in the MRT, then the Agentinitialization is not interrupted and no MRT is loaded. Ifan error is detected when a configuration refresh isrequested, then the refresh is not performed.

AOFSPOOLFULLCMD 1 SA z/OS does not execute the Spool recovery passesmore than once. Message AOF2941I is issued if theSPOOLFULL condition persists.

0 SA z/OS re-executes the Spool recovery commands.


AOFSPOOLSHORTCMD 1 SA z/OS does not execute the Spool recovery passesmore than once. Message AOF2941I is issued if theSPOOLSHORT condition persists.

0 SA z/OS re-executes the Spool recovery commands.


Global Variables


|



AOFSTATUSCMDSEL 0 Issue all status commands or replies that are associatedwith the new status, without respect to any specifiedselection values. No thresholds are checked for theminor resource subsystem.status to derive selectioncriteria or prevent the issuing of commands or replies ifcritical thresholds are exceeded.

If AOFSTATUSCMDSEL is not set, or it is set to a valueother than 0, only commands or replies with a givenselection criterion such as starttype or stoptype areissued.

AOFUPDAM Yes or No This controls whether updates are made in theautomation manager.

No is the default setting.

AOFUPDRODM Yes or No This controls whether updates are made in RODM andmust be set to the same value for each system within asysplex.

No is the default setting.

AOFUSSWAIT 1 to n This is the number of seconds SA z/OS waits for thecompletion of a user-defined z/OS UNIX monitoringroutine (specified in the z/OS UNIX ControlSpecification panel) until it receives a timeout. When thetimeout occurs, SA z/OS no longer waits for a responsefrom the monitoring routine and sends a SIGKILL to themonitoring routine.


INGCICS_CORRWAIT User-definednumeric value

The number of seconds that INGCICS waits for outputfrom a CICS transaction. If not specified, INGCICS usesa default CORRWAIT (CCDEF) value.

INGIMS_CORRWAIT User-definednumeric value

The number of seconds that INGIMS waits for outputfrom an IMS command. If not specified, INGIMS usesthe default CORRWAIT (CCDEF) value.

INGOPC_MULTIPLIER 1 to n This is used in conjunction with AOFRMTCMDWAITand AOFRPCWAIT to determine how long to waitbefore giving up.

INGPAC_SHOWNOLIMIT 1 Pacing gates that have a concurrency limit of NOLIMITare always shown on the INGPAC display. This is thedefault.

0 Pacing gates that have a concurrency limit of NOLIMITare not shown on the INGPAC display.

INGRAITF_WAIT User-definednumeric value

The number of seconds that the INGRAITF routinewaits.

INGREQ_ORIGINATOR 1 Indicates that SA z/OS assigns individual originatorIDs for each operator issuing an INGREQ command.

0 All operators are grouped under originator IDOPERATOR.


Global Variables


|||||

|||

Parameter Defaults for CommandsTable 26. Global Variables That Define the Installation Defaults for Specific Commands

Variable Name Description Reference 1

AOFSETSTATEOVERRIDE Sets the default OVERRIDE value for the SETSTATE command. SETSTATE

AOFSETSTATESCOPE Allows you to override the predefined default for the SCOPEparameter of the SETSTATE command.

SETSTATE

AOFSETSTATESTART Allows you to override the predefined default for the STARTparameter of the SETSTATE command.

SETSTATE

DISPEVT_WAIT Sets the WAIT parameter of the DISPEVT command to the specifiedvalue.

DISPEVT

DISPEVTS_WAIT Sets the WAIT parameter of the DISPEVTS command to the specifiedvalue.

DISPEVTS

DISPGW_COLUMNS Specifies the panel column layout of the DISPGW command.

This is a stem where:

v DISPGW_COLUMNS.0 contains the number of elements

v DISPGW_COLUMNS.n contains the definition of the n-th column

Each column definition is a blank separated list of tokens in thesequence:

v prefix is the prefix indicator of the column which can be:

I when the column is 'immovable' which implies 'prefixed'

P when the column is 'prefixed' for horizontal scrolling

F when the column is 'floating' (that is, scrolled)

- when the column is hidden

v sortorder is the sort order of the column which can be:

A when the column is sorted in ascending order

D when the column is sorted in descending order

- when the column is not sorted or hidden

v sortkey is the sort key of the column which can be:

1-n when the column is sorted in the n-th place


v columnname is the name of the column. These can be multiplewords. Because this name is used to reference the column youcannot change column names.

TGLOBALS with that name are a user-specific setting and theyoverwrite the installation defaults. It is recommended to save a layoutto TGLOBALS and then derive CGLOBALS when needed. Invaliddefinitions are silently ignored.

DISPGW

Global Variables


|

|||

||||

||||

||||

||||

||

|

|

|

||

|

|

|

|

|

|

|

|

|

|

|

|

|||

||||

|

Table 26. Global Variables That Define the Installation Defaults for Specific Commands (continued)


DISPMTR_COLUMNS Specifies the panel column layout of the DISPMTR command.


v DISPMTR_COLUMNS.0 contains the number of elements

v DISPMTR_COLUMNS.n contains the definition of the n-th column
















DISPMTR

Global Variables


|

|||

||

|

|

|

||

|

|

|

|

|

|

|

|

|

|

|

|

|||

||||

|



DISPSTAT_COLUMNS Specifies the panel column layout of the DISPSTAT command.


v DISPSTAT_COLUMNS.0 contains the number of elements

v DISPSTAT_COLUMNS.n contains the definition of the n-th column
















DISPSTAT

DISPTRG_WAIT Sets the WAIT parameter of the DISPTRG command to the specifiedvalue.

DISPTRG

Global Variables


|

|||

||

|

|

|

||

|

|

|

|

|

|

|

|

|

|

|

|

|||

||||

|

||||



EVIRYDP0_COLUMNS Specifies the panel column layout of the INGIMS REQ=DEPcommand.


v EVIRYDP0_COLUMNS.0 contains the number of elements

v EVIRYDP0_COLUMNS.n contains the definition of the n-th column
















INGIMS

Global Variables


|

|||

|||

|

|

|

||

|

|

|

|

|

|

|

|

|

|

|

|

|||

||||

|



INGAMS_COLUMNS Specifies the panel column layout of the INGAMS command.


v INGAMS_COLUMNS.0 contains the number of elements

v INGAMS_COLUMNS.n contains the definition of the n-th column
















INGAMS

INGAUTO_INTERVAL Sets the default for the INTERVAL parameter of the INGAUTOcommand.

INGAUTO

INGEVENT_WAIT Sets the WAIT parameter of the INGEVENT command to the specifiedvalue. The parameter specifies whether or not to wait until the requestis complete.

INGEVENT

INGEXEC_RESP Sets the RESP parameter of the INGEXEC command to the specifiedvalue.

INGEXEC

INGEXEC_SELECT Sets the SELECT parameter of the INGEXEC command to the specifiedvalue.

INGEXEC

INGEXEC_TIMEOUT Sets the TIMEOUT parameter of the INGEXEC command to thespecified value.

INGEXEC

INGEXEC_WAIT Sets the WAIT parameter of the INGEXEC command to the specifiedvalue.

INGEXEC

INGGROUP_WAIT Sets the WAIT parameter of the INGGROUP command to the specifiedvalue. The parameter specifies whether or not to wait until the requestis complete.

INGGROUP

INGHIST_MAX Sets the MAX parameter of the INGHIST command to the specifiedvalue.

INGHIST

INGHIST_WIMAX Sets the WIMAX parameter of INGHIST command to the specifiedvalue.

INGHIST

Global Variables


|

|||

||

|

|

|

||

|

|

|

|

|

|

|

|

|

|

|

|

|||

||||

|

||||

||||

|

||||

||||

||||

||||

||||

|

||||

||||



INGIMS_CMDWAIT Sets the CMDWAIT parameter (the maximum wait time for acommand to complete) of the INGIMS command to the specifiedvalue.

INGIMS

INGIMS_REQ Sets the REQ parameter (the request to be issued to the IMSsubsystem) of the INGIMS command to the specified value.

INGIMS

INGINFO_WAIT Sets the WAIT parameter of the INGINFO command to the specifiedvalue.

INGINFO

INGLKUP_TIMEOUT Sets the TIMEOUT parameter of the INGLKUP command to thespecified value.

INGLKUP

INGLKUP_WAIT Sets the WAIT parameter of the INGLKUP command to the specifiedvalue.

INGLKUP

INGLIST_COLUMNS Specifies the panel column layout of the INGLIST command.


v INGLIST_COLUMNS.0 contains the number of elements

v INGLIST_COLUMNS.n contains the definition of the n-th column
















INGLIST

INGLIST_WAIT Sets the WAIT parameter of the INGLIST command to the specifiedvalue.

INGLIST

INGMON_WAIT Sets the WAIT parameter of the INGMON command to the specifiedvalue.

INGMON

INGMOVE_WAIT Sets the WAIT parameter of the INGMOVE command to the specifiedvalue.

INGMOVE

INGPAC_WAIT Sets the WAIT parameter of the INGPAC command to the specifiedvalue.

INGPAC

INGRELS_SHOW Sets the SHOW parameter of the INGRELS command to the specifiedvalue.

INGRELS

Global Variables


|

|||

||||

|

||||

||||

||||

||||

||

|

|

|

||

|

|

|

|

|

|

|

|

|

|

|

|

|||

||||

|

||||

||||

||||

||||

||||



INGRELS_WAIT Sets the WAIT parameter of the INGRELS command to the specifiedvalue.

INGRELS

INGREQ_EXPIRE Sets the default EXPIRE parameter of the INGREQ command to thespecified value.

INGREQ

INGREQ_INTERRUPT Sets the default INTERRUPT parameter of the INGREQ command tothe specified value. The parameter specifies whether or not theautomation manager should wait until the resource has reached its UPstate, but the resource is still in the startup phase when the higherpriority stop request is given.

INGREQ

INGREQ_OVERRIDE Sets the default OVERRIDE parameter of the INGREQ command tothe specified value.

INGREQ

INGREQ_PRECHECK Sets the default PRECHECK parameter of the INGREQ command tothe specified value.

INGREQ

INGREQ_PRI Sets the default priority (PRI parameter) of the INGREQ command tothe specified value.

INGREQ

INGREQ_PRI.E2EMGR Specifies the priority that incoming requests from the end-to-endautomation manager are executed at. Default: LOW

INGREQ

INGREQ_REMOVE Sets the default value for the REMOVE parameter of the INGREQcommand to the specified value. It the resource reaches the specifiedstatus (condition), the request is automatically removed.

INGREQ

INGREQ_REMOVE.START Sets the default value for the REMOVE parameter of the INGREQSTART command. If not specified the value set by INGREQ_REMOVEis used.

INGREQ

INGREQ_REMOVE.STOP Sets the default value for the REMOVE parameter of the INGREQSTOP command. If not specified the value set by INGREQ_REMOVEis used.

INGREQ

INGREQ_RESTART Sets the default for the RESTART parameter of the INGREQ commandwhen shutting down the resource.

INGREQ

INGREQ_SCOPE Sets the SCOPE parameter of the INGREQ command to the specifiedvalue.

INGREQ

INGREQ_SOURCE Sets the default SOURCE parameter of the INGREQ command to thespecified value. The parameter specifies the originator of the request.

INGREQ

INGREQ_TIMEOUT Sets the interval in minutes used to check for the INGREQ commandused to check whether the request has been successfully completed,and whether to send a message or cancel the request if it has not beensatisfied after that time.

INGREQ

INGREQ_TYPE Sets the default startup/shutdown type (TYPE parameter) of theINGREQ command to the specified value.

INGREQ

INGREQ_VERIFY Sets the default VERIFY parameter of the INGREQ command to thespecified value.

INGREQ

INGREQ_WAIT Sets the WAIT parameter of the INGREQ command to the specifiedvalue.

INGREQ

INGRPT_WAIT Sets the WAIT parameter of the INGRPT command to the specifiedvalue.

INGRPT

INGRUN_CMT Sets the CMT parameter of the INGRUN command to the specifiedvalue.

INGRUN

INGRUN_MULT Sets the MULT parameter of the INGRUN command to the specifiedvalue.

INGRUN

Global Variables


|

|||

||||

||||

||||||

|

||||

||||

||||

||||

||||

|

||||

|

||||

|

||||

||||

||||

|||||

|

||||

||||

||||

||||

||||

||||



INGRUN_OVERRIDE Sets the OVERRIDE parameter of the INGRUN command to thespecified value.

INGRUN

INGRUN_PERSISTENT Sets the PERSISTENT parameter of the INGRUN command to thespecified value.

INGRUN

INGRUN_PRI Sets the PRI parameter of the INGRUN command to the specifiedvalue.

INGRUN

INGRUN_REQ Sets the REQ parameter of the INGRUN command to the specifiedvalue.

INGRUN

INGRUN_RUNMODE Sets the RUNMODE parameter of the INGRUN command to thespecified value.

INGRUN

INGRUN_RUNRES Sets the RUNRES parameter of the INGRUN command to thespecified value.

INGRUN

INGRUN_SYSTEM Sets the SYSTEM parameter of the INGRUN command to the specifiedvalue.

INGRUN

INGRUN_TARGET Sets the TARGET parameter of the INGRUN command to the specifiedvalue.

INGRUN

INGRUN_TYPE Sets the TYPE parameter of the INGRUN command to the specifiedvalue.

INGRUN

INGRUN_VERIFY Sets the VERIFY parameter of the INGRUN command to the specifiedvalue.

INGRUN

INGRUN_WAIT Sets the WAIT parameter of the INGRUN command to the specifiedvalue.

INGRUN

INGSCHED_WAIT Sets the WAIT parameter of the INGSCHED command to the specifiedvalue. The parameter specifies whether or not to wait until the requestis complete.

INGSCHED

INGSET_VERIFY Sets the default VERIFY parameter of the INGSET command to thespecified value.

INGSET

INGSET_WAIT Sets the WAIT parameter of the INGSET command to the specifiedvalue. The parameter specifies whether or not to wait until the requestis complete.

INGSET

INGSTX_WAIT Sets the WAIT parameter of the INGSTX command to the specifiedvalue.

INGSTX

INGTRIG_WAIT Sets the WAIT parameter of the INGTRIG command to the specifiedvalue.

INGTRIG

INGVOTE_EXCLUDE Sets the EXCLUDE parameter of the INGVOTE command to thespecified value. The parameter specifies the resource types (forexample SVP or GRP) to be excluded when showing all requests.Resources of that type are filtered out.

INGVOTE

INGVOTE_SHOW Sets the SHOW parameter of the INGVOTE command to the specifiedvalue.

INGVOTE

INGVOTE_SOURCE Sets the default SOURCE parameter of the INGVOTE command to thespecified value.

INGVOTE

INGVOTE_STATUS Sets the STATUS parameter of the INGVOTE command to thespecified value. The parameter specifies which requests should bedisplayed: winning, losing or all.

INGVOTE

INGVOTE_WAIT Sets the WAIT parameter of the INGVOTE command to the specifiedvalue.

INGVOTE

1. See the specified command in IBM Tivoli System Automation for z/OS Operator's Commands.

Global Variables


|

|||

||||

||||

||||

||||

||||

||||

||||

||||

||||

||||

||||

||||

|

||||

||||

|

||||

||||

|||||

|

||||

||||

||||

|

||||

|

Global Variables


Appendix B. Customizing the Status Display Facility

Overview of the Status Display FacilityThis information explains how to customize the Status Display Facility (SDF)panels, descriptors, and operations.

How the Status Display Facility WorksThe SA z/OS Status Display Facility (SDF) uses colors and highlighting torepresent subsystem resource states.

Typically, a subsystem shown in green on the SDF status panel indicates it is up,while red indicates a subsystem in a stopped or problem state. SDF can be tailoredto present the status of system components in a hierarchical manner.

Types of SDF PanelsFigure 52 on page 266 shows several SDF screens for system CHI01. This figureshows the main types of panels used in SDF:v The root component

v The status component

v The detail status display

In addition to these panel types, you can create other types of panels according toyour system requirements and the applications you are monitoring.


Root ComponentThe root component is typically an element appearing on the first screen displayedwhen SDF is started. In Figure 52, the CHI01 system is the root component.

Status ComponentResources monitored by SDF are called status components. In Figure 52, systemCHI01 has JES2, RMF™, VTAM, TSO, and NetView status components, as shownon the CHI01 System Status panel. The status component panel displays allmonitored resources in a system. Each monitored resource is shown in the color ofits current status. For example, JES2 is shown in green if it is up.

Detail Status DisplayA detail status display is built from information in a status descriptor (see “StatusDescriptors”). This panel is displayed by tabbing to the appropriate resource onthe status component panel and pressing the detail PF key. Each status componentcan have one or more status descriptors, or detail records, associated with it.

Figure 52 shows an example detail status display for a JES2 status descriptor. The 1of 3 on the panel indicates that JES2 currently has three status descriptors, andtherefore three detail status displays, associated with it.

Status DescriptorsA status descriptor is a detailed record of information about a resource status. In itsraw form, a status descriptor is a multiline SA z/OS message containinginformation such as:v Root component and status component to which the status descriptor applies

DATA CENTER SYSTEMS

CHI01

===>

1=HELP

CHI01 SYSTEM STATUS

===>

1=HELP

JES2

RMF

VTAM

TSO

NETVIEW

--- DETAIL STATUS DISPLAY ---

===>

1=HELP 3=RETURN 6=ROLL 7=UP 8=DOWN

COMPONENT : JES2 SYSTEM : CHI01

COLOR : GREEN PRIORITY : 550

DATE : 09/08/90 TIME : 09:02:17

REPORTER : GATACOS6 NODE : CHI01

REFERENCE VALUE: JES2

AOF571I 08:59 : JES2 SUBSYSTEM STATUS FOR

JOB JES2 IS UP - AT NETVIEW INITALIZATION

1 OF 3

Root Component(System Panel)

Status Component(Monitored Resources)

Detail Status Display

Figure 52. Example SDF Panels

Overview of the Status Display Facility


v Priority, color, and highlighting associated with the status descriptor (see “HowStatus Descriptors Affect SDF” on page 268 for more information)

v Date and time the status descriptor was generatedv Actual resource status information; for example, an SA z/OS message indicating

the resource is up

SDF uses information in a status descriptor to generate a detail status display (see“Detail Status Display” on page 266). You do not usually look directly at a statusdescriptor; rather, you look at portions of it through a detail status display. Forexample, in Figure 52 on page 266, the detail status display presents informationfrom a status descriptor for status component JES2. The 1 of 3 on the panelindicates that JES2 currently has three status descriptors associated with it.

SDF generates, displays, and deletes status descriptors.

SDF Tree StructuresSDF uses tree structures to set up the hierarchy of monitored resources displayed onSDF status panels. An SDF tree structure usually starts with the system name asthe root node and has a level number of one. Tree structure levels subordinate tothe root node are the monitored resources. The level numbers of these resourcesreflect their dependency on each other.

The whole tree structure is kept in a data space for performance reasons. The sizeof the data space really depends on your usage of SDF. Refer to"MAXTREEDSPSZ" in IBM System Automation for z/OS Programmers Reference forcalculating and defining the size of the data space.

You define SDF tree structures in NetView DSIPARM data set member AOFTREE.

Figure 53 shows an example SDF tree structure. Figure 52 on page 266 shows howthese statements result in a tree structure.

SA z/OS supplies a sample SDF tree structure in the SA z/OS sample library. Thistree structure is referenced by a %INCLUDE statement in member AOFTREE in theNetView DSIPARM data set. You can customize this sample tree structure to meetyour requirements. This order of dependency does not have to be the same as thatused for system startup or shutdown using SA z/OS. System symbols aresupported for the tree structure. This can help reduce both customization work anderrors.

1 SY12 SYSTEM

3 WTOR3 APPLIC

4 AOFAPPL5 AOFSSI

4 JES4 VTAM

3 TSO3 RMF

2 GATEWAY2 MONITOR2 APG

3 GROUP

Figure 53. Example SDF Tree Structure


Appendix B. Customizing the Status Display Facility 267

For example, using the tree structure in Figure 53 on page 267, if there is a problemwith TSO, it is not desirable to also change the VTAM status color, because VTAMis not having any problems. In contrast, in the SA z/OS startup and shutdownprocedures, TSO is dependent on VTAM.

More details on SDF tree structure definitions are in “Step 1: Defining SDFHierarchy” on page 275.

How Status Descriptors Affect SDFStatus descriptors are the main units of information SDF uses. The information instatus descriptors determines how your SDF status displays look at any point intime. This section explains how SDF uses status descriptors.

Priority and Color AssignmentsStatus descriptors are assigned both a priority number and a color. These color andpriority assignments determine the colors in which status components aredisplayed. In SDF, a lower number indicates a higher priority. Status descriptorsare connected to the status component in ascending order of priority.

Color and priority assignments for status descriptors are defined in two places:v in the PRIORITY parameter in the AOFINIT member of the NetView DSIPARM

data set. This parameter defines initial priority and color assignments used forstatus descriptors. The values defined in AOFINIT are used if no furthercustomization is done to priority and color assignments. The default priorityranges and colors used in AOFINIT are:

Priority Range Color

001 to 199 Red

200 to 299 Pink

300 to 399 Yellow

400 to 499 Turquoise

500 to 599 Green

600 to 699 Blue

White is used as the default status descriptor color (the DCOLOR parameter inmember AOFINIT, described in IBM Tivoli System Automation for z/OS Programmer'sReference) and as the default color for a status component without a tree structureentry (the ERRCOLOR parameter in member AOFINIT, described in IBM TivoliSystem Automation for z/OS Programmer's Reference). For more information on thePRIORITY parameter, see IBM Tivoli System Automation for z/OS Programmer'sReference.v in the SDF definitions in the Status Details policy object. These entries define

colors, highlighting, and priorities used for particular resource statuses. Colorand priority assignments defined in the customization dialog can be used tooverride assignments in the AOFINIT member.

Note: Some of the resource statuses that appear in SDF displays do not directlycorrespond to resource statuses used in the automation status file.IBM Tivoli System Automation for z/OS Programmer's Reference shows the defaultresource status types, colors, highlighting, and priorities provided withSA z/OS. These settings define to SA z/OS the parameters used when addingstatus descriptors to SDF.



||

For more information on the SDF Status Details definition, see “Step 4:(Optional) Defining SDF in the Customization Dialog” on page 280.

Chaining of Status Descriptors to Status ComponentsA resource status change causes a status descriptor to be generated. SDF adds thisstatus descriptor to a chain of status descriptors. Chained status descriptorsdetermine the status and color of status components. The highest-priority statusdescriptor in a chain determines the initial color in which the status component isdisplayed. The underlying chained priority numbers determine the color thatsuccessive detail status displays are shown in.

Status descriptors are chained off each level of status component in a treestructure. Status descriptors chained to lower-level status components are alsochained to a higher-level status component, again in order of priority. Statusdescriptors are also chained off the root component. These status descriptors are allthe status descriptors that currently exist at all levels of the tree structure.

For example, Figure 54 shows status descriptors currently generated for systemSY1. The priority for each status descriptor is shown by a number.

The status components at the lowest level in this tree structure, JES2, RMF, andVTAM, have status descriptors chained off them. Status component JES2 has threestatus descriptors chained, with priorities 1, 10, and 50. Because 1 is the highestpriority, the status descriptor with priority 1 is organized first in the chain. Thishighest-priority status descriptor determines the color in which JES2 is displayedon the status panel. If an operator uses the detail PF key to view detail statusdisplays for JES2, the information contained in the status descriptor with priority 1is displayed first, then the detail status display for the status descriptor withpriority 10, and so on.

At the SYSTEM status component level in the tree structure, all status descriptorsfrom the lower-level status components are also chained. Because the statusdescriptors chained to RMF and VTAM have higher priorities than the priority 10and 50 status descriptors for JES2, they are organized after the priority 1 status

100 50 10 7 5 1

50 10 7 5 1

1 SY1

100

1

10

50

5 7

2 SYSTEM 2 GATEWAY

3 JES2 3 RMF 3 VTAM

Root Component

Status descriptorfor GATEWAYstatus component

Status descriptorfor VTAM statuscomponent

Status descriptorfor RMF statuscomponent

Status descriptorsfor JES2 statuscomponent

Status descriptors for SY1 root component

Status descriptors for SYSTEM status component

Figure 54. Status Descriptors Chained to Status Components



descriptor in the chain. An operator using the detail PF key at the SYSTEM levelcould view five detail status displays, ranging from priority 1 to priority 50.

Similarly, at the SY1 level in the tree structure, all status descriptors chained to allstatus components in the tree structure are chained in order of priority. Anoperator using the detail PF key at the SY1 level could view six detail statusdisplays, ranging from priority 1 to priority 100.

If a status component has multiple status descriptors with equal priorities, thestatus descriptors are chained off the status component in order of arrival time.

When a status descriptor no longer accurately reflects the actual status of aresource, SDF automatically deletes it from status descriptor chains. As an exampleof how priority determines order of status descriptors, suppose two statusdescriptors currently exist for status component JES2. If there are two statusdescriptors for JES2 with priorities of 120 and 140, the status descriptor withpriority 120 is displayed first. In both cases, JES displays in red on the SDF statuspanel.

In SA z/OS, all statuses are defined in the automation control file. When anautomation event occurs, the SA z/OS AOCUPDT command scans the automationcontrol file for the SDF entry for that status. SA z/OS issues a request to add thestatus using the information from the automation control file.

For example, suppose subsystem RMF, shown on the example SDF panels inFigure 52 on page 266, is set to a STOPPING state. The SA z/OS AOCUPDTcommand scans the automation control file for the STOPPING state entry for SDFand generates a status descriptor, specifying a priority of 420. SDF adds the statusdescriptor to the RMF status component. RMF appears as yellow and reverse onthe status panel. Once RMF is in a stopped state, the AOCUPDT command scansthe automation control file for the STOPPED state SDF entry and generates a statusdescriptor with priority 120. SDF adds this new status descriptor to the RMF statuscomponent. Now, RMF appears in red on the SDF status panel.

Propagating Status Descriptors Upward and Downward in a TreeStructureBased on the order of dependencies defined in a tree structure, status descriptorscan be propagated upward or downward to status components in a tree structure.

This propagation of status descriptors affects the color in which status componentsare displayed, as well as the detail status displays operators can view by using thedetail PF key on a particular status component.

Propagation of status upward and downward in a tree structure is defined by thePROPUP and PROPDOWN parameter in the AOFINIT member (see IBM TivoliSystem Automation for z/OS Programmer's Reference for descriptions).

The SA z/OS-provided defaults for status propagation in the AOFINIT memberare to propagate status upward (PROPUP=YES) but not downward(PROPDOWN=NO).

When status is propagated upward in a tree structure, if a status descriptor isadded or deleted at a lower level in the tree structure, it is also added or deletedfrom the cumulative chain of status descriptors at a higher-level node in the treestructure.



|||||||||

Propagation of status upward in a tree structure consolidates the status of allmonitored resources in the system at the root node. In this way, the color of theroot node reflects the most important or critical status in a computer operationscenter. For example, in Figure 53 on page 267, any color changes for AOFSSI arereflected in AOFAPPL, APPLIC, SYSTEM, and SY1, if SDF propagates statuschanges upward in the tree structure. In Figure 52 on page 266, if all monitoredresources are green, the root node CHI01 on the Data Center Systems panel is alsoshown in green.

When status is propagated downward in a tree structure, if a status change occursat a higher level in a tree structure, the changes are sent downward in the treestructure. This propagating downward could cause status descriptors at lowerlevels in the tree structure to be added or deleted.

Propagating status downward can be useful when an entire system is down. Insuch a case, you want SDF status panels to accurately reflect the system status. Youdo not want status components lower in the tree structure to retain previouslygenerated status descriptors indicating that the components are up and running,because these status descriptors do not accurately reflect the status of thecomponents. You can configure your SDF implementation to propagate statusdownward, and remove all status descriptors from all status components in a treestructure. If an operator tries displaying detailed status about any of the statuscomponents lower in the tree structure, they receive "NO DETAIL INFOAVAILABLE" messages. The empty chain color, defined by the EMPTYCOLORparameter in member AOFINIT with a default color of blue, is also used toindicate that no detail information is available. See IBM Tivoli System Automation forz/OS Programmer's Reference for the EMPTYCOLOR description.

How SDF Helps Operations to Focus on Specific ProblemsSDF structure and processing allows the program identifying a problem to beconcerned only with the specific problem.

For example, suppose an application program detects a warning message for statuscomponent JES on CHI01. The following processing steps occur:1. The application program issues a request to SDF to add a status descriptor for

JES.2. The status entry for JES on system CHI01 now indicates there is a problem

with JES. If the SDF is configured to propagate status up the hierarchical treestructure, the status for system CHI01 also reflects the problem state. See IBMTivoli System Automation for z/OS Programmer's Reference for details on thePROPUP SDF initialization parameter.

3. Now, suppose another more serious problem occurs. The application programwhich detects this new problem issues another request to SDF to add a statusdescriptor having a lower priority number than the status descriptor for thefirst problem.

4. Because status descriptors are chained in order of priority, the JES status nowreflects the status descriptor color of the more serious problem.

5. When the more serious problem is resolved, the application program detectingthe problem resolution issues a request to SDF to remove the status descriptorfor this problem from the chain of JES status descriptors.

6. The status panel is updated to reflect the first problem.



How SDF Panels Are DefinedAll SDF status panels, apart from detail status display panels, are defined in theAOFPNLS member of the NetView DSIPARM data set.

Member AOFPNLS can contain either one or both of the following:v %INCLUDE statements referencing other NetView DSIPARM members

containing definitions of panels. The %INCLUDE statement causes the namedpanel definition member to be loaded. This is the recommended method, andthe method used in the SA z/OS-provided version of AOFPNLS. Systemsymbols are supported for the %INCLUDE statements. This can help reduceboth customization work and errors.

v Panel structure definitions for all SDF panels.

Panel members defined or referenced in AOFPNLS are loaded into systemmemory, and may be deleted, replaced, or temporarily made resident using theSDFPANEL command (see IBM Tivoli System Automation for z/OS Programmer'sReference for command description).

Panels that are to be dynamically loaded as needed (see “Dynamically LoadingTree Structure and Panel Definition Members”) must be defined in a NetViewDSIPARM member having the same member name as the panel itself.

It is recommended that you include only frequently used panels in AOFPNLS, toconserve system memory. Other panels can be dynamically loaded when needed,either by pressing a SDF function key or by using the SCREEN command.

Note: Dynamic refresh only works with panels that are defined in AOFPNLS.

SDF internally formats and builds detail status display panels from the informationin a status descriptor. You do not have to define and format detail status displaypanels. Status components defined in the panel definitions must also be defined inthe corresponding tree structure. However, not all status components defined inthe tree structure require a corresponding entry on the SDF status panel. Forexample, in Figure 53 on page 267, the APPLIC status component is only apseudo-entry and may not actually be displayed on any SDF status display panel.

SDF status panels can be customized to reflect any environment. For example, youcan define a panel to show the status of all JES subsystems on all processors in acomputer operations center. The JES operator can view the panel to determine thestatus of any JES subsystem in the complex.

For detailed information on defining SDF panels, see “Step 2: Defining SDFPanels” on page 276.

Dynamically Loading Tree Structure and Panel DefinitionMembers

Using %INCLUDE statements in the main SDF tree structure and panel definitionmembers allows you to dynamically load tree structure and panel definitionmembers without restarting SDF

See IBM Tivoli System Automation for z/OS Programmer's Reference. The SDFTREEcommand loads a tree structure definition member. The SDFPANEL commandloads a panel definition member. You can dynamically reload members AOFTREEand AOFPNLS themselves.



The RESYNC SDFDEFS command generates the SDF panels using the advancedautomation option (AAO) AOF_AAO_SDFROOT.n for the SDF root names that areto be applied. (See IBM Tivoli System Automation for z/OS Operator's Commands). Italso replicates SDF tree definitions that have defined variables beginning with&SDFCxxx using the corresponding AAO AOF_AAO_SDFCxxx.n. Config RefreshMonitoring uses the reserved AAO AOF_AAO_SDFCsaplex.n.

Note: Both members AOFPNLS and AOFTREE must not contain any variable thatis subject to replication. (See "AOFTREE" and "Status Component Panel Definition"in IBM Tivoli System Automation for z/OS Programmer’s Reference).

Refer to "Status Component Panel Definition" in IBM Tivoli System Automation forz/OS Programmer's Reference for the setup of AOF_AAO_SDFROOT.n variables.

Refer to "AOFTREE" in IBM Tivoli System Automation for z/OS Programmer'sReference for the setup and usage of AOF_AAO_SDFCxxx.n variables.

Using SDF for Multiple SystemsYou can configure SDF so that multiple systems in an automation network canforward their resource status information to the SDF on the focal point system.

In a multiple-system environment, the following must be defined:v The tree structure for each system must be defined in the AOFTREE member of

NetView DSIPARM on the focal point system SDF. The root name must beunique for each system tree structure.

v For target system SDF status update to occur on a focal point SDF, SA z/OSfocal point services must already be implemented.

Because each root name must be unique in a multiple-system environment, anystatus component on any system defined to the focal point SDF can be uniquelyaddressed by prefixing the status component with the root component name:ROOT_COMPONENT.STATUS_COMPONENT

For example:SY1.JES2

Similarly, any SDF status descriptors forwarded from the target system to the focalpoint SDF are prefixed with the root name of the target system by SA z/OSroutines.

SDF ComponentsSDF consists of the following components:

Table 27. SDF Components

Name Type Purpose

AOFTDDF Task Initializes SDF and maintains the status database. Thisinitialization is an automated function.

SDF Command Starts an SDF operator session.

SDFTREE Command Dynamically loads or deletes an SDF tree structuredefinition member from the NetView DSIPARM data set.

SDFPANEL Command Dynamically loads or deletes an SDF panel definitionmember from the NetView DSIPARM data set.



||||||

|

Table 27. SDF Components (continued)

Name Type Purpose

AOFINIT Input file Contains SDF initialization parameters defined with thestatements described in IBM Tivoli System Automation forz/OS Programmer's Reference. AOFINIT is in the NetViewDSIPARM data set.

AOFTREE Input file Contains tree structures described in IBM Tivoli SystemAutomation for z/OS Programmer's Reference. This memberusually consists of a list of %INCLUDE statementsreferencing other members containing tree structures.AOFTREE is in the NetView DSIPARM data set.

AOFPNLS Input file Contains SDF panel parameters defined by the statementsdescribed in “Step 2: Defining SDF Panels” on page 276.This member usually consists of a list of %INCLUDEstatements referencing other members containing paneldefinitions. AOFPNLS is in the NetView DSIPARM dataset.

panel_name Input file A DSIPARM member containing the definition of one ormore SDF panels or %INCLUDE statements identifyingother DSIPARM panel definition members. It is highlyrecommended that panel definition members contain thedefinition of a single panel having the same name as themember.

tree_name Input file A DSIPARM member containing the definition of one ormore tree structures. It is highly recommended that treedefinition members contain the definition of a single treehaving the same root component name as the membername.

How the SDF Task Is Started and StoppedDuring SA z/OS initialization, the AOFTDDF task loads members defining panelformat, panel flow, and tree structures.

Member AOFINIT defines parameters common to all SDF panels and basicinitialization specifications, such as screen size, default PF keys, and the initialscreen displayed when a SDF session is started. These AOFINIT parameters aredescribed in IBM Tivoli System Automation for z/OS Programmer's Reference.

Starting the SDF TaskIn SA z/OS code, the AOFTDDF task is started by the following command:START TASK=AOFTDDF

Stopping the SDF TaskIn SA z/OS code, the AOFTDDF task is stopped by the following command:STOP TASK=AOFTDDF

Note: When SDF is restarted, all existing SDF status descriptors are lost, as theyare kept only in memory.

SDF Definition ProcessUse the following procedure to define the panels displayed in an SDF session.



Details on each step are provided later in this chapter and in IBM Tivoli SystemAutomation for z/OS Programmer's Reference.1. Define the hierarchy of monitored resources used for your SDF panels, using

tree structure statements in NetView DSIPARM data set members. These treestructure definition members should be referenced by %INCLUDE statementsin the main SDF tree structure definition member, AOFTREE, in the NetViewDSIPARM data set. See IBM Tivoli System Automation for z/OS Programmer'sReference for details.

2. Define SDF status panels using panel definition statements in NetViewDSIPARM data set members. Panels can either be automatically loaded whenSDF starts, or dynamically loaded using the SDFPANEL command. For panelsto be automatically loaded, add %INCLUDE statement specifying the paneldefinition member to the main panel definition member, AOFPNLS, in the(conref) DSIPARM data set. See “Step 2: Defining SDF Panels” on page 276 fordetails. Define and customize SDF status panels in the following general order:a. Root panelb. Status component panel for each entry on the root panelc. Any other customized status panels.

3. Customize the SDF initialization parameters in NetView DSIPARM memberAOFINIT, if necessary (optional), or use defaults. See IBM Tivoli SystemAutomation for z/OS Programmer's Reference for detailed descriptions of SDFinitialization parameters. Using defaults is recommended.

4. Define SDF resource status, color, highlight and priority values using thecustomization dialog to edit the SDF Status Display policy object, or usedefaults. This step is optional. See IBM Tivoli System Automation for z/OSDefining Automation Policy for the description of the Status Display policyobject. Using defaults is recommended.

Notes:

a. Resources that SA z/OS is not currently automating are not displayed onSDF panels.

b. To display the status of multiple systems and forward status from targetsystems to SDF on a focal point system, SA z/OS focal point services mustalready be implemented. See IBM Tivoli System Automation for z/OS DefiningAutomation Policy for details on configuring focal point services.

Step 1: Defining SDF HierarchyMember AOFTREE in the NetView DSIPARM data set contains a set of definitionsthat define the propagation hierarchy for status color changes.

When the status changes for a component, the corresponding color change ispropagated up or down the tree to the next higher or lower level component. Thelevel is determined by the level number assigned to each component. The type ofpropagation is determined either by the entry in the AOFINIT member or byindividual requests to add a status descriptor to a status component.

Note: SA z/OS does not use this SDF hierarchy for subsystem shutdown orstartup procedures. Instead, SA z/OS uses subsystem entries defined in theautomation policy to determine startup and shutdown relationships andhierarchies.

Tree Structure DefinitionsAOFTREE contains tree structure definitions. To define tree structures, you can:

SDF Definition Process


v Use %INCLUDE statements that reference other members containing definitionsfor specific tree structures. This is the recommended method, and the methodused in the SA z/OS-provided version of AOFTREE.On the %INCLUDE statement, the name of the referenced member must beenclosed in parentheses.

v Place all tree structure definitions in AOFTREE.v Use a combination of both.

System symbols are supported wherever they are used in the AOFTREE, AOFINITand AOFPNLS members. This can help reduce both customization work anderrors.

Figure 55 shows a typical tree structure definition:

In this tree structure, SY1 is the root component. This definition is in a separatemember, named SY1. It is referenced by the following statement in the AOFTREEmember:%INCLUDE(SY1TREE)

Loading Tree Structures:

All tree structures need not be loaded during initialization. Some can be loadeddynamically after SDF is started. To do this, use AOFTREE to define those treestructure entries that are loaded during initialization, then use the SDFTREEcommand to load additional tree structures as needed.


Tree structures loaded after SDF is started must be contained in separate members.Each member must be named after the root component for which the tree structureis defined.

Step 2: Defining SDF PanelsSDF status panels are defined in NetView DSIPARM member AOFPNLS.

SA z/OS loads the panel definitions in AOFPNLS when SDF is initialized.

Panel Definition MethodsTo define panels in AOFPNLS, you can:

1 SY12 SYSTEM

3 WTOR3 APPLIC

4 AOFAPPL5 AOFSSI

4 JES4 VTAM

3 TSO3 RMF

2 GATEWAY2 MONITOR2 APG

3 GROUPS

Figure 55. Example Tree Structure Definition

SDF Definition Process


v Use %INCLUDE statements referencing separate NetView DSIPARM memberscontaining panel definitions. This is the recommended method, and the methodused in the SA z/OS-provided version of AOFPNLS. See “%INCLUDEStatement for SDF Panels” on page 279 for details on using the %INCLUDEstatement for SDF panel definition members.

v Include actual definitions for all panels.v Use a combination of both %INCLUDE statements and panel definitions.v Include a subset of panel entries to load during initialization, so that additional

panel definitions can be loaded only when needed (see IBM Tivoli SystemAutomation for z/OS Programmer's Reference).


Panel Definition StructureThe structure of each panel definition is as follows:v Begin panel definition statement (PANEL)v Status component definition statements, consisting of pairs of the following

statements:– STATUSFIELD: defines location of a status component on a panel– STATUSTEXT: defines the text displayed in the STATUSFIELD

v Text fields and data definition statements, consisting of pairs of the followingstatements:– TEXTFIELD: defines locations and attributes for constant fields on panels– TEXTTEXT: defines text displayed in the TEXTFIELD

v Status panel PF key definitions (PFKnn)You should assign the SDFCONF command to the PF4 key. Use the followingdefinition:PFK4=SDFCONF &ROOT,&COMPAPPL,&RV,&SID,&SNODE,&DATE,&TIME,&DA

Using SDFCONF to delete a record in SDF is useful because it prompts you forconfirmation before performing the actual deletion. If you do not want theprompt panel to appear, then add ",VERIFY=NO" to the end of the SDFCONFcommand.You must call SDFCONF to delete exceptional messages, that is, capturedmessages with the severity Unusual, Important and Critical. The SDFCONFcommand removes a message entry from the SDF control structure and alsofrom all other interfaces where the message is shown, for example, TEP andNMC.

v End panel statement (ENDPANEL)

Descriptions of these panel definition statements are in IBM Tivoli SystemAutomation for z/OS Programmer's Reference.

Recommended Order for Defining PanelsWhen defining panels, it is recommended that you define them in the followingorder:1. The root panel2. The status components for each item listed on the root panel3. Any other customized status panels


Note: This order of defining panels is a recommendation only. You can define yourSDF panels in any order desired.

Example Panel DefinitionFigure 56 shows how an example SDF panel looks when it is displayed.

Figure 57 shows the panel definition statements required to define the panel inFigure 56.

In Figure 57, the panel name is SYSTEM. This panel definition can either be in aseparate member referenced by an %INCLUDE statement in AOFPNLS or bedirectly coded in AOFPNLS. The recommended method is to use a separatemember and an %INCLUDE statement. If it is in a separate member, the membername is SYSTEM. You do not have to explicitly define every PF key for the panel.PF key definitions not specified are picked up from definitions in NetViewDSIPARM member AOFINIT.

Table 28 on page 279 describes each statement in Figure 57:

SYSTEM DATA CENTER SYSTEMS

SY1 GATEWAY

===>1=HELP 2=DETAIL 3=RET 6=ROLL 7=UP 8=DN 10=LF 11=RT 12=TOP

Figure 56. Example SDF Panel

PANEL(SYSTEM,24,80)TEXTFIELD(01,02,10,WHITE,NORMAL)TEXTTEXT(SYSTEM)TF(01,25,57,WHITE,NORMAL)TT(DATA CENTER SYSTEMS)STATUSFIELD(SY1,04,04,11,N,,SY1SYS)STATUSTEXT(SY1)SF(SY1.GATEWAY,02,40,47,N,,GATEWAY)ST(GATEWAY)TF(24,01,79,T,NORMAL)TT(1=HELP 2=DETAIL 3=RET 6=ROLL 7=UP 8=DN ,10=LF 11=RT 12=TOP)PFK1(AOCHELP SDF)PFK2(DETAIL)PFK3(RETURN)PFK6(ROLL)PFK7(UP)PFK8(DOWN)PFK10(LEFT)PFK11(RIGHT)PFK12(TOP)ENDPANEL

Figure 57. Example Panel Definition Entry


Table 28. Panel Definition Entry Description

Statement Description and Example Value

PANEL(SYSTEM,24,80) The panel definition statement. The panel name is SYSTEM, thepanel length is 24, and the panel width is 80.

TEXTFIELD(01,02,10,WHITE,NORMAL) The text location statement defining constant panel fields. This fieldstarts on line 01 in position 02 and ends in position 10. The color ofthe field is white and highlighting is normal.

TEXTTEXT(SYSTEM) The text data statement specifying the actual data that goes in thetext field just defined. This field contains the word SYSTEM.

TEXTFIELD and TEXTTEXT are always grouped in pairs.

TF(01,25,57,WHITE,NORMAL) Another TEXTFIELD statement for another constant field.

TT(DATA CENTER SYSTEMS) Another TEXTTEXT statement for the text field just defined.

STATUSFIELD(SY1,04,04,11,N,,SY1SYS) The location of the status component field. The status component isSY1. This field starts on line 04 in position 04 and ends in position11. The highlighting level is normal. The next panel displayed whenthe Down PF key is pressed is SY1SYS.

STATUSTEXT(SY1) The text data used for the name of the field just defined with theSTATUSFIELD statement. In this case, the field name is SY1.

STATUSFIELD and STATUSTEXT statements are grouped in pairs.

SF(SY1.GATEWAY,02,40,47,N,,GATEWAY) Another STATUSFIELD definition.

ST(GATEWAY) Another STATUSTEXT definition.

TF(24,01,79,T,NORMAL)TT(1=HELP 2=DETAIL 3=RET 6=ROLL 7=UP,8=DN 10=LF 11=RT 12=TOP)

Here, TEXTFIELD and TEXTTEXT are used to display PF keydefinitions. For this panel, these are the default definitions definedin AOFINIT. If you need values differing from the defaults, there isa statement for defining PF keys unique to this panel, DPFKnn. SeeIBM Tivoli System Automation for z/OS Programmer's Reference for adescription of this statement.

PFK1(AOCHELP SDF)PFK2(DETAIL)PFK3(RETURN)PFK6(ROLL)PFK7(UP)PFK8(DOWN)PFK10(LEFT)PFK11(RIGHT)PFK12(TOP)

PF key definition statements.

ENDPANEL The end panel statement, indicating that this is the end of definitionsfor this panel.

%INCLUDE Statement for SDF PanelsThe %INCLUDE statement for SDF has the following features:v The SDF %INCLUDE statement allows the specification of a list of members

rather than a single member only. Each member name in the list represents aDSIPARM member that is to be loaded. Member names in the list are delimitedby a comma.

v The SDF %INCLUDE statement requires parentheses around the specifiedmember or members.

v You can specify the option STATIC or DYNAMIC for the SDF %INCLUDEstatement. If you specify DYNAMIC, this generates the panel definitions for all


of the system names that you specify in AOF_AAO_SDFROOT.n common globalvariable (see Table 25 on page 242). STATIC is the default.

v The target DSIPARM members may contain only complete panel definitions oradditional %INCLUDE statements. Panel definitions must be contained within asingle member, and therefore cannot be built using commonly defined segments.


Step 3: (Optional) Customizing SDF Initialization ParametersMember AOFINIT allows you to define parameters common to all SDF panels andSDF initialization specifications, such as:v Initial screen shown when SDF is startedv Maximum operator logon limitv Default PF key definitionsv Detail status display panel PF key definitionsv Detail status display panel PF key descriptionsv Default priorities and colors

These parameters define values for SDF when it is started.


This step of SDF customization is optional. Using SA z/OS-provided defaultvalues for these parameters is recommended.

Note: User-defined statuses are not saved across a recycle or a monitor cycle. Thismeans the status of a subsystem changes from the user-defined status to anappropriate SA z/OS status.

Step 4: (Optional) Defining SDF in the Customization DialogThe SDF entries in the Status Display policy object allow you to define statusesand the priorities assigned to those statuses.

These entries are used by SA z/OS commands to gather data for requests to addstatus descriptors to status components. The format and values used in SDF StatusDetail definitions are described in IBM Tivoli System Automation for z/OSProgrammer's Reference.

This step of SDF customization is optional. Using SA z/OS-provided definitionsfor SDF is recommended.


|

Appendix C. Coordinating System Operations with AutomaticRestart Manager

SA z/OS system operations provides coordination with the Automatic RestartManager. The Automatic Restart Manager (ARM) is a base z/OS component. It is arecovery function that automatically restarts designated applications when:v The application ends abnormally.v The system that the application is running on is part of a sysplex, and that

system fails. In this case, ARM will attempt to restart the application on anothersystem within the sysplex.

SA z/OS coordinates with ARM to:v Determine which facility is responsible for restarting a specific application.v Avoid possible duplications or conflicts in application recovery attempts.v Allow you to take full advantage of SA z/OS fallback capabilities for

applications running on sysplexes. SA z/OS continues to automate anapplication after it has been moved to a fallback system, provided SA z/OS isinstalled on that system. If it is not installed on the fallback system, SA z/OS isstill aware that the application is active on a system other than its primary oneand does not attempt to restart it.

You have to define the Automatic Restart Manager policy using the administrativedata utility for ARM policy data (IXCMIAPU) described in z/OS MVS Setting Up aSysplex.

SA z/OS resolves Automatic Restart Manager statuses to SA z/OS statuses,incorporates Automatic Restart Manager-related conditions, and provides onestatus related to Automatic Restart Manager:v EXTSTART - The application is being started or restarted externally.

Defining an ARM Element NameAutomatic Restart Manager uses element names to identify the applications withwhich it works. Each Automatic Restart Manager enabled application must have aunique element name for itself that it uses in all communication with AutomaticRestart Manager.

Automatic Restart Manager tracks the element name and has its policy defined interms of element names. If an application moves between systems it MUSTcontinue to use the same element name as it did on the original system. For moreinformation on defining Automatic Restart Manager names to SA z/OS, see“Application Entry Type” in IBM Tivoli System Automation for z/OS DefiningAutomation Policy.

All Automatic Restart Manager elements are unregistered initially. Transitionsbetween statuses are caused by:v IXCARM macro invocationsv Application failuresv System failuresv Timeouts


A minor resource definition subsystem.0ARM can be used to tailor automationbehaviour during ARM restart processing. As an example, a subsystem.0ARM minorresource could be specified with a RESTART EXIT enabled to drive a user suppliedexit during ARM restart. The user exit would control additional actions to be takenduring ARM restart of the subsystem. If the RESTART flag for this minor resourceis resolved to 'N', SA z/OS will not allow ARM to attempt a restart of theapplication.

Rather than use the subsystem.0ARM minor resource definition, a RESTART EXITcould also be specified against the major resource definition for the application. Inthis case the exit would be driven for all application restarts, not just ARM.

Other reasons for SA z/OS not to allow ARM to attempt a restart of theapplication are:v The application's monitor indicates that the address space is already active.v The application is involved in a shutdown.v The application is in status BREAKING, BROKEN, or CTLDOWN.

Defining a MOVE Group for Automatic Restart ManagerAll resources with the same ARM element name should be linked to one SysplexApplication Group of nature MOVE (MOVE group).

An application's ARM element name is defined either during creation on theDefine New Entry panel for applications or after creation via policy itemAPPLICATION INFO, in both cases in the MVS Automatic Restart ManagementElement Name field.

In order to ensure an application in a MOVE group has completely deregisteredfrom ARM before the automation manager attempts to restart it, aPrepareavailable/WhenObservedDown (passive) relationship must be defined foreach ARMed application in the MOVE group with the MOVE group defined as thesupporting resource.

To make sure that the automation manager will start the applications linked to aMOVE group, the applications should not be in a HardDown status. The Start OnIPL option should not be set to NO.

For more information on how to define MOVE groups see “Creating a NewApplicationGroup” in IBM Tivoli System Automation for z/OS Defining AutomationPolicy.

Defining an ARM Element Name


Appendix D. Message Automation

Generic Synonyms: AOFMSGSYThis AOFMSGSY NetView automation table (AT) fragment contains a number ofsynonyms that must be appropriately set. It is used in most master automationtables to set up the environmental parameters for the other fragments. TheAOFMSGSY member is supplied by SA z/OS (in the SINGNPRM data set).

You must customize it for each of your systems. The customized copy should beplaced in the domain-specific data set for that system.

Note that many values in this table fragment are enclosed in triple single quotationmarks. This means that the value of the synonym is the value entered surroundedby a single set of single quotation marks. This is necessary so that the value istreated as a literal and not an automation table variable.

Synonym Usage and Default

%AOFALWAYSACTION% This synonym contains the action statement used for all themessages within a Begin-End block that SA z/OS does nottrigger any action for.

Default: NULL

The default is that no action is taken and the message doesnot continue to search for further matches within the sameAT.

%AOFDOM% This synonym should contain the domain ID of the SA z/OSNetView on the system that it is automating. The synonym isused to screen messages to prevent the SA z/OS on thismachine from reacting to a message that originated onanother machine. If not set correctly, your automation fails.

Default: &DOMAIN.

This is a default domain name used in a number of thesamples.

%AOFSYS% This synonym should contain the system name used in thelast IPL of the system. It is used to screen messages toprevent the SA z/OS on this machine from reacting toevents that have occurred on other machines. It is importantif you are running on a JES3 global or in a sysplex withEMCS consoles. If not set correctly, your automation fails.

Default: &SYSNAME.

This is a default system name used in a number of thesamples.

%AOFARMPPI% This synonym should contain the name of the NetViewautotask that is running the PPI interface from SA z/OS toz/OS. It is used to route commands from the NetViewautomation table to the autotask.

Default: AOFARCAT



%AOFGMFHSWAIT% The time interval SA z/OS waits after GMFHS initializationis complete before issuing the command to update theRODM with the current application automation states.Following the issuing of message DUI4003I GMFHSNETWORK CONFIGURATION INITIALIZEDSUCCESSFULLY, GMFHS resets the color of all SA z/OSicons to grey (unknown). To set the SA z/OS icons' color tothe current automation states after the initialization ofGMFHS, SA z/OS must wait and issue the updatecommand AFTER GMFHS has reset the colors to grey.

Default: 00:02:00

SA z/OS Message Presentation: AOFMSGSYThe presentation of SA z/OS messages (prefixed with AOF, ING, HSA, EVJ, EVEand EVI) under NetView is controlled by the automation table. This uses a numberof synonyms and task globals indicating your message display characteristics. Thefollowing synonyms determine the display characteristics for each type of message.There is one set for the normal presentation of the message (AOFNORMx) and asecond set for the held presentation (AOFHOLDx).


%AOFHOLDI% This synonym defines the actions taken for SA z/OSinformation (type I) messages that are held on your NCCFconsole.

Default: HOLD(Y) COLOR(GRE) XHILITE(REV)

This:

v Ensures that the message is held

v Causes the message to be displayed in reverse video green

%AOFHOLDA% This synonym defines the actions taken for SA z/OSimmediate action (type A) messages that are held on yourNCCF console. As a rule, you should specify HOLD(Y) inthe action.

Default: HOLD(Y) COLOR(RED) XHILITE(REV) BEEP(Y)

This:


v Causes the message to be displayed in reverse video red

v Sounds the terminal alarm when the message is displayed

%AOFHOLDD% This synonym defines the actions taken for SA z/OSdecision (type D) messages that are held on your NCCFconsole. As a rule, you should specify HOLD(Y) in theaction.

Default: HOLD(Y) COLOR(WHI) XHILITE(REV) BEEP(Y)

This:


v Causes the message to be displayed in reverse video white


Generic Synonyms: AOFMSGSY



%AOFHOLDE% This synonym defines the actions taken for SA z/OSeventual action (type E) messages that are held on yourNCCF console. As a rule, you should specify HOLD(Y) inthe action.

Default: HOLD(Y) COLOR(YEL) XHILITE(REV) BEEP(Y)

This:


v Causes the message to be displayed in reverse videoyellow


%AOFHOLDW% This synonym defines the actions taken for SA z/OS waitstate (type W) messages that are held on your NCCFconsole. As a rule, you should specify HOLD(Y) in theaction.

Default: HOLD(Y) COLOR(PIN) XHILITE(REV) BEEP(Y)

This:


v Causes the message to be displayed in reverse video pink


%AOFNORMI% This synonym defines the actions taken for SA z/OSinformation (type I) messages that are not held on yourNCCF console. As a rule, you should not specify HOLD(Y)in the action.

Default: COLOR(GRE)

This:

v Ensures that the message is not held

v Causes the message to be displayed in green

%AOFNORMA% This synonym defines the actions taken for SA z/OSImmediate Action (type A) messages that are held on yourNCCF console. As a rule, you should not specify HOLD(Y)in the action.

Default: COLOR(YEL) XHILITE(REV) BEEP(Y)

This:


v Causes the message to be displayed in yellow


%AOFNORMD% This synonym defines the actions taken for SA z/OSDecision (type D) messages that are held on your NCCFconsole. You may find it beneficial to force these messages tobe held.

Default: COLOR(WHI) XHILITE(BLI)

This:


v Causes the message to be displayed in blinking white

SA z/OS Message Presentation: AOFMSGSY

Appendix D. Message Automation 285


%AOFNORME% This synonym defines the actions taken for SA z/OSEventual Action (type E) messages that are not held on yourNCCF console. As a rule, you should not specify HOLD(Y)in the action.

Default: COLOR(YEL)

This:

v Ensures that the message is not held

v Causes the message to be displayed in yellow

%AOFNORMW% This synonym defines the actions taken for SA z/OS WaitState (type W) messages that are held on your NCCFconsole. You may find it beneficial to force these messages tobe held.

Default: HOLD(Y) COLOR(PIN) XHILITE(REV) BEEP(Y)

This:


v Causes the message to be displayed in reverse video pink


Operator Cascades: AOFMSGSYThe next set of synonyms defines a series of operator cascades. A cascade is basicallya list of automation operators used in many of the fragments to route commands.If %CASCADE% is defined as a synonym for ’AUTMON AUTBASE AUTO1’ and you route acommand to it with ROUTE (ONE %CASCADE%) on an EXEC statement, the command isrun on the first autotask in the cascade that is logged on. This provides you with aflexible, controllable means of providing backup processing tasks in case one ofyour normal tasks is unavailable.


%AOFLOPAUTOx% This cascade defines the actions taken for SA z/OSinformation (type I) messages that are being held on yourNCCF console. Given the number of informational messagesthat SA z/OS produces you may find it beneficial HOLD(N)to stop them from being held even if the user has asked forthem to be held.

Default: ’’AUTOx’’

%AOFOPAUTO1% This cascade is used to route commands to AUTO1. If youhave renamed AUTO1 you must change the synonym.

Default: AUTO1

There is no backup for AUTO1. If it fails when it is needed,many other things will probably fail as well.

%AOFOPAUTO2 This cascade is used to route commands to AUTO2. If youhave renamed AUTO2 you must change this synonym.

Default: AUTO2 AUTO1

If AUTO2 is not active, AUTO1 does its work.

SA z/OS Message Presentation: AOFMSGSY



%AOFOPBASEOPER% This cascade is used to send commands to BASEOPER. Ifyou are not using the standard names for SA z/OSautotasks you must change this synonym. BASEOPER ismainly defined as a fallback operator and has very littlework directly routed to it.

Default: AUTBASE AUTO1

AUTBASE is the operator ID that SA z/OS uses forBASEOPER in its other samples. If AUTBASE is not active,AUTO1 is tried.

%AOFOPRPCOPER% This cascade is used for XCF communication management. Ifyou are not using the standard names for SA z/OSautotasks you must change this synonym.

Default: AUTRPC AUTSYS AUTBASE AUTO1

%AOFOPSYSOPER% This cascade is used to send commands to SYSOPER. If youare not using the standard names for SA z/OS autotasksyou must change this synonym. SYSOPER is mainly definedas a fallback operator and has very little work directlyrouted to it.

Default: AUTSYS AUTBASE AUTO1

AUTSYS is the operator ID that SA z/OS uses for SYSOPERin its other samples.

%AOFOPMSGOPER% This cascade is used to send commands to MSGOPER. If youare not using the standard names for SA z/OS autotasksyou must change this synonym. MSGOPER is mainlydefined to respond to miscellaneous messages.

Default: AUTMSG AUTSYS AUTBASE AUTO1

AUTMSG is the operator ID that SA z/OS uses forMSGOPER in its other samples.

%AOFOPNETOPER% This cascade is used to send commands to NETOPER. If youare not using the standard names for SA z/OS autotasksyou must change this synonym. NETOPER is defined forVTAM automation.

Default: AUTNET1 AUTNET2 AUTSYS AUTBASE AUTO1

AUTNET1 and AUTNET2 are the operator IDs that SA z/OSuses for NETOPER in its other samples. NETOPER is theonly sample automation function to have a backup definedin the samples.

%AOFOPJESOPER% This cascade is used to send commands to JESOPER. If youare not using the standard names for SA z/OS autotasksyou must change this synonym. JESOPER is mainly definedfor JES automation.

Default: AUTJES AUTSYS AUTBASE AUTO1

AUTJES is the operator ID that SA z/OS uses for JESOPERin its other samples.

Operator Cascades: AOFMSGSY



%AOFOPMONOPER% This cascade is used to send commands to MONOPER. Ifyou are not using the standard names for SA z/OSautotasks you must change this synonym. MONOPER isused for regular monitoring and subsystem startups.

Default: AUTMON AUTSYS AUTBASE AUTO1

AUTMON is the operator ID that SA z/OS uses forMONOPER in its other samples.

%AOFOPRECOPER% This cascade is used to send commands to RECOPER. If youare not using the standard names for SA z/OS autotasksyou must change this synonym. RECOPER is used forrecovery processing.

Default: AUTREC AUTSYS AUTBASE AUTO1

AUTREC is the operator ID that SA z/OS uses forRECOPER in its other samples.

%AOFOPSHUTOPER% This cascade is used to send commands to SHUTOPER. Ifyou are not using the standard names for SA z/OSautotasks you must change this synonym. SHUTOPERcoordinates automated shutdowns.

Default: AUTSHUT AUTSYS AUTBASE AUTO1

AUTSHUT is the operator ID that SA z/OS uses forSHUTOPER in its other samples.

%AOFOPGSSOPER% This cascade is used to send commands to GSSOPER. If youare not using the standard names for SA z/OS autotasksyou must change this synonym. GSSOPER is used forgeneric subsystem automation.

Default: * AUTGSS AUTSYS AUTBASE AUTO1

AUTGSS is the operator ID that SA z/OS uses for GSSOPERin its other samples.

If you want to turn off the "ASSIGN BY JOBNAME" feature,that is, the advanced automation CGLOBAL variableAOF_ASSIGN_JOBNAME (see “Global Variables” on page199) has been set to 0, you must remove the asterisk (*),because this may cause serialization problems.Note: The NetView command ASSIGN-BY-JOBNAME thatoccurs before automation table processing only affectsmessages that are associated with an MVS job name.

%AOFOPWTORS% This cascade is used to route commands concerning WTORS.If you are not using the standard names for SA z/OSautotasks you must change this synonym. Its use ensuresthat all WTOR processing is done on the same task and thisis serialized.

Default: * AUTGSS AUTSYS AUTBASE AUTO1

This specifies that AUTSYS is to do all the WTORprocessing.




%AOFOPGATOPER% This cascade is used to route commands to this domain'sgateway autotask. Because the autotask name contains thedomain ID you must modify this synonym.

Default: GATRdomain.

AOF01 is the default domain used in the other samples.There is no backup as the gateway CLISTs expect to berunning on GATOPER.

SA z/OS Topology Manager for NMC: AOFMSGSTThese synonyms are used and defined in the AOFMSGST fragment.


%AOFOPTOPOMGR% This is the name of the autotask that the SA z/OS topologymanager runs on this system.

Default: &DOMAIN.TPO

%AOFINITOPOCMD% This is the command issued to initialize the SA z/OStopology manager.

Default: INGTOPO INIT &DOMAIN.TPO

%AOFOPHB% This is the name of the heart beat task needed on focal point.

Default: AUTHB



SA z/OS Topology Manager for NMC: AOFMSGST


Appendix E. TSO User Monitoring

Active TSO users can be monitored in NMC and SDF using the SA z/OScommand DFTSOU (EVJETSOU).

To enable TSO user monitoring add the following entry to user AT includefragment INGMSGU1 (or to your own user message table):IF (MSGID=’IEF125I’ | MSGID=’IEF126I’ | MSGID=’IEF450I’)

THEN EXEC(CMD(’DFTSOU UPDATE’) ROUTE(ALL *))DISPLAY(N) NETLOG(N) CONTINUE(Y);

Also, put 'DFTSOU SCAN' in the STARTUP-REFRESHSTART policy for the TSOsubsystem.

When DFTSOU is called with the UPDATE parameter then:v For IEF125I, an ADD request is sent to SDF and NMC for the TSO user that

produces the message.v For IEF126I, a DELETE request is sent to SDF and NMC for the TSO user that

produces the message.v For IEF450I, a DELETE request is sent to SDF and NMC for the failing TSO user.

When IEF450I is specified, and the trap is coded in INGMSGU1, thenCONTINUE(Y) must also be coded.

When DFTSOU is called with the SCAN parameter, an MVS D TS,L command isissued to identify all currently active TSO users. This data is then passed to SDFand NMC.

NMC updates are associated with NMC object TSO. SDF updates are associatedwith SDF tree entry TSOUSERS.


||

Appendix F. Autodiscovery Mapper Files and Report Formats

Mapper FilesThe mapping files all conform to the same set of syntax rules.v The first line is a version indicator.v Comments are prefixed by a hash ('#') sign.v Data lines are tabular, with the layout of the columns being fixed for each

mapping file. If the data gets out of alignment the mappings will not work.v The columns in each mapping file are split into input and output columns. The

input columns are used to search the data table, and the resulting outputcolumns are applied to the data. In most of the files only the first match is used– and the user data is searched before the System Automation (SA) data. Amatch in the user data will prevent the SA data from being searched. Theexception is the variable mapping file, where all matching data lines areprocessed and the SA data is processed before the user data. This allows theuser data to overwrite values generated by the SA data.

v The values specified in the columns can be exact matches or can use thefollowing wildcards:– An asterisk ('*') will match any number of characters, including none.– A percentage sign ('%') will match a single, required character.– You can use a backward slash ('\') as an escape character, causing the

following wildcard character to be treated as a literal character – so \* wouldmatch an * in the input data.

For example, to match a string of 4 or more characters followed by IPC, youwould need to specify %%%%*IPC. The four percent signs require that there areat least 4 characters before the IPC, the * matches whatever character(s), if any,come next and the 'IPC' constant is required at the end. It will match ABCDIPC,ABCDEIPC or ABCDEFGHIPC.

v In almost all cases the width of the input column is the maximum length of thedata value. The exceptions are single character values where you need to use anescape character. These are 2 characters wide to allow for the escape characterand the match value (a '\*').

FunctionsSome mapping files support the entry of formulas into their output columns.When entering these you need to follow these rules:v Functions and constants follow REXX syntax, and you can use the full set of

REXX functions.v Constants need to be delimited by single quotes.v Symbols can be substituted in using &symbol. syntax. For example

&JOBNAME.'@'&AOCCLONE9. would produce the jobname for the addressspace followed by an @ followed by the value of &AOCCLONE9 – for example,NET@IPCSD.

v There are some columns that support the entry of formulas with unresolved&AOCCLONEx. values. These values get resolved later at SA load time, andallow the same APL to be used with &AOCCLONE values from differentsystems, dynamically generating different values. When entering a formula intothese, care must be taken, as you cannot use a REXX function on an unresolved


&AOCCLONEx. value. The workaround is to establish the value you want touse in the variable mapping file and use that in the formula in the policy file.Note however that the value will be fixed and not dynamic like the&AOCCLONEx. values.

Address Space Identification Mapping File ( INGSMAID /INGSMAIU)

This file contains rules to identify address spaces from their discovery data.

To see data about the Address Spaces that have been discovered and how theyhave been identified, see the “ASDETAIL Report” on page 303.

Table 29. Input Columns for Address Space Identification Mapping File

Input Column Number of Characters Values Used

SYSPLEX 8 chars

SYSTEM 8 chars

AS TYPE 2 chars A, I, J, M, S, U, * or blank

JOB PREFIX 1 chars T, S, J or blank

JOB NAME 8 chars

PROC NAME 8 chars

STEP NAME 8 chars

USERID 8 chars

CMD PREFIX 8 chars

SUBSYS NAME 8 chars

PRIMARY SUBSYS 1 chars P or blank

ASC SUBSYS NAME 8 chars

SCHED SUBSYS NAME 8 chars

PREVIOUS ID 24 chars

PROG NAME 8 chars

PROG PARMS 100 chars

Table 30. Output columns for Address Space Identification Files

Column Name Number of Characters

NEW ID 24 chars

The data searched against the table is the raw discovery data before any otheridentification has been performed. The PREVIOUS ID value will either be a valuefrom the discovery engine or blank. The value in the NEW ID column overwritesthe PREVIOUS ID value for the Address Space.

Table 31. Job Prefix Values

Value Job name

J Job

S Started Task

T TSO User

blank Unknown

Mapper Files


Table 32. Address Space Type Values

Value Address Space Type

A ATX (APPC Transaction)

I Initiator

J Job

M Mount job

S Started Task

U TSO User

* System (use \* to match)

XCF Group Identification Mapping File ( INGSMGRP /INGSMGRU)

This file contains rules to identify established XCF Groups.

To see details of the XCF Groups discovered and how they have been identified,see the “XCFGROUP Report” on page 301.

Table 33. Input Columns for XCF Group Identification Mapping File

Input ColumnName Number of Characters

SYSPLEX 8 chars

SYSTEM 8 chars

XCF GROUP NAME 8 chars

Table 34. Output Columns for XCF Group Identification Mapping File

Column Name Number of Characters

GROUP TYPE 24 chars

While the SA provided file contains identification rules for some applications withhard-coded XCF Group names – SA itself, for example, always names its maingroup INGXSGxx – there are a number of other applications – such as TivoliWorkload Scheduler – that leave the entire construction of the name up to you.Entries for such applications need to be made in your user mapping file, reflectingthe naming conventions used within the target sysplex. The GROUP TYPE valueslisted in the SA provided file are the only SA defined values in the groupidentification space. Currently they only have to correspond with GROUP TYPEentries in the XCF Group Member Mapping files.

XCF Group Member Identification Mapping File (INGSMGMB /INGSMGMU)

This files contains rules for identifying members of XCF groups.

To see details of the XCF Groups and their Members that were discovered andhow they have been identified, refer to the “XCFGROUP Report” on page 301.

Mapper Files

Appendix F. Autodiscovery Mapper Files and Report Formats 295

Table 35. Input Columns for XCF Group Member Identification Mapping File

Input Column Name Number of Characters

SYSPLEX 8 chars

SYSTEM 8 chars

XCF GROUP TYPE 24 chars

XCF MEMBER NAME 16 chars

XCF MEMBERJOBNAME

8 chars

Table 36. Output Columns for XCF Group Member Identification Mapping File

Output Column Name Number of Characters

Address Space Type 24 chars

The XCF GROUP TYPE comes from the XCF GROUP identification mapping file.The Address Space ID replaces any previous identification of the address space bythe discovery engine or the Address Space Identification mapping file. In somecases, simple membership of an XCF Group implies the nature of the members. Inother cases, there is a specific naming convention that can be used to determine therole. Trackers for Tivoli Workload Scheduler may join the group with a TRK suffix,while controllers may join with a CTL suffix, but this is up to your local namingconventions. In some cases the XCF group member name is the only way todistinguish the application instances.

USS Process Identification Mapping File (INGSMUID /INGSMUIU)

This file contains rules for identifying USS Processes. It is applied after the AddressSpace Identification rules and the XCF Group Member Identification rules.

The USS data for the processes has been linked to the z/OS aspect of thoseprocesses through matching ASID values. To see the USS Process details that havebeen discovered, look in the “ASDETAIL Report” on page 303.

Table 37. Input Columns for USS Process Identification Mapping File


SYSPLEX 8 chars

SYSTEM 8 chars

JOB NAME 8 chars

z/OS USERID 8 chars

USS USERID 8 chars

PREVIOUS ID 24 chars

z/OS PROG NAME 8 chars

USS COMMAND 256 chars

Table 38. Output Columns for USS Process Identification Mapping File


NEW ID 24 chars

Mapper Files


The z/OS values come from the automated Address Space data rather than fromthe discovered USS data. This file is useful for identifying applications that rununder USS, many of which can be indistinguishable just by looking at the z/OSside of the discovered data.

Policy Mapping File (INGSMPLU / INGSMPLY)This file contains records that perform three tasks.

The first is to select which Address Spaces are to be modelled in SA policy andwhich are not. This is controlled by the Model Location field. A value of X willexclude the address space from the model, while values of U or S will cause it tobe included. The second is to select the model that will be used to represent theaddress space. This is done via the Model Entry Type and Model Entry Namefields. At present only entries of type APL may be modelled. If these fields areblank, no model will be used and just a basic APL entry will be created for theaddress space (which will require manual completion before it can be usedsuccessfully for automation). If they are not, they indicate an APL entry in eitherthe SA KB (Model Location: S) or the User KB (Model Location: U) that will beused as a model for the APL entry that is created for the address space. The thirdis to provide non-default values for some attributes of the APL entry that will becreated. Note that the Jobname and ARM Element Name fields support delayedresolution of &AOCCLONE values in their formula. You may not use REXXfunctions on &AOCCLONE values in these fields.

Table 39. Input Columns for Policy Mapping File

Input Column Name Number of characters

SYSPLEX 8 chars

SYSTEM 8 chars

ADDRESS SPACE ID 24 chars

JOBNAME 8 chars

PROCNAME 8 chars

Table 40. Output Columns for Policy Mapping File

Output ColumnName Number of characters Values Used

MODELLOCATION

1 char S, U, or X

MODEL ENTRYTYPE

3 chars APL

MODEL ENTRYNAME

20 chars

ENTRY NAME 40 chars (Formula - 20 chars)

AUTOMATIONNAME

40 chars (Formula - 11 chars)

JOBNAME 40 chars (Formula - 8 chars*)

ARM ELEMENTNAME

40 chars (Formula - 16 chars*)

APL TYPE 12 chars

APL SUBTYPE 12 chars

CLASS 40 chars (Formula - 20 chars)

Mapper Files


The Entry name defaults first to the jobname. If this is not unique, it is suffixedwith the sysplex name. If this is still not unique, it is suffixed with a sequentialnumber. If you specify a specific entry name it will still be subject to the sameuniqueness checking as the jobname. If you do not like the entry name you can usethe customization dialog facilities to rename it. This will not cause a second copyto appear if you rerun the discovery, preload and import process.

The Automation Name defaults to the jobname, because this is the name youroperators will use to interact with the application under SA's control and it is aname that they are already familiar with. The jobname should be unique on eachsystem, so we should not get naming clashes.

The jobname defaults to the jobname, but we allow you to enter a formula for it ifyou wish to add an AOCCLONE value into it. This will not be reflected in theAutomation Name. The formula must evaluate to the discovered value.

The ARM Element Name also defaults to the discovered value and, again, we haveprovided you with a way of overriding it with a formula including an&AOCCLONE value. The formula must evaluate to the discovered value.

Some Address Space specific symbols are available to use in the formulas:&JOBNAME.&PROCNAME.&CMDPREFIX.&ARM_ELEMENT.&SCHED_SUB.

If you open up the SA KB, you can inspect the APL models that SA provides. Donot change the contents of this PDB as it can be updated by APAR, which wouldoverwrite your changes. To create your own user models, you need to define yourUser KB PDB and create/copy entries into it. SA provides a sample PDB you canuse to seed it which contains a class for a generic APL that is automated by IEF403and 404 messages only. While it is far from the best model for most APLs, it is agood starting point. You can also copy and modify samples from the SA KB PDB.If you copy an entry into the User KB PDB, you should copy the entry from theSA Policy Mapping File into your User Policy Mapping File and edit it to point tothe model you have created in your User KB.

The use of classes in the KB is strongly recommended and SA will copy them overif a model requires them – but note that once they have been copied, it will notrecopy them. This is to protect any changes that you may have manually made tothe class after it was copied out of the KB.

Variable Mapping File (INGSMVRS / INGSMVRU)This file contains rules defining symbols that can be used in formulas.

Table 41. Input Columns for Variable Mapping File


SYSPLEX 8 chars

SYSTEM 8 chars

Mapper Files


Table 42. Output Columns for Variable Mapping File


VAR NAME 16 chars

VAR FORMULA 48 chars

Note that any variables called AOCCLONE, or AOCCLONEx (where x is 0-9, A-Z)will end up in the customization dialogs set as an AOCCLONE value on thesystem being discovered.

Preloader ReportsThe reports generated by the preloader are:

SUMMARYAn overview of the import process.

XCFGROUPDetails of XCF groups found and identified, along with members foundand identified. You need the data from this report to construct your ownXCF Group and Member identification records.

ARMGROUPDetails of Automatic Restart Groups and Elements that were discovered.

ASDETAILDetails of Address Spaces found and Identified, along with all of theirdiscovery data. You need the information from this report to constructyour own Address Space identification and Policy mapping records.

EXCLUDEThis contains the results of running the identified address spaces throughthe policy mapping file. It shows which address spaces have been includedin the policy and which have been excluded.

CONSTRCTThis contains information about how the flatfile was constructed. For eachaddress space it will indicate whether or a new APL was constructed or anexisting one was relinked. It indicates which sample policy (if any) wascopied from the KB and which classes (if any) were also copied.

SYMBOLSThis contains information about the list of symbolic values that werediscovered on the target system, along with data about any variablesdefined in the variable mapping files.

KBIMPORTThis contains information about which entries were imported from the SAand User KBS.

KBMAPThis is an 'index' for the SA and User KBs, listing which entries are inwhich KB.

SUBSYSThis lists the z/OS subsystems that were identified, which address space isproviding them, which other address spaces are associated with it andwhich address spaces are being scheduled under it. The following sectiondescribes the reports in detail. Note that it is not possible for you to addadditional reports or to customize the existing reports.

Mapper Files


Summary ReportThis report contains basic information summarizing the preloader process –elapsed times for various steps and counts of things discovered, excluded,included and so forth. The phases of the discovery data import are:

Parameter ReviewThe first phase resolves the mapping files and the report that will be used.

Discovery ImportThe raw snapshot file is read in, the variable and ID mapping files areconsulted and the address spaces are identified. It generates theASDETAIL, XCFGROUP and SYMBOLS reports.

PDB ImportThe extract of the Target PDB is read in and analyzed. Entries for the readentities are output into the CONSTRCT report.

Generation PhaseThe policy mapping file is read and applied. This is where the decision ismade as to whether an Address Space is a new APL or a previouslyknown one. It produces the EXCLUDE report and the second part of theCONSTRCT report.

KB ImportThe extracts of the two KB PDBs are read in and analyzed.

KB XrefA second pass through the KB data to resolve dependencies betweeninstances.

Output PhaseThe flat file is assembled and output. This produces the second part of theCONSTRCT report.

The SUMMARY report gives an overview of activity in these phases.

Preloader Reports


XCFGROUP ReportThis report contains details of the XCF groups located, their identification, theassociated members and their identification. Note that only members on the systembeing modelled will be identified in the report.

System Automation for z/OS V3R5 Discovery Import--------------------------------------------------ParametersSA Mapping file..........: KEY.DRIVER.AINGIMAPInstallation mapping file: KEY.DRIVER.AINGIMAPReport File..............: MIK.DISCO.REPORT.LOAD1

Generated: 24 Feb 2014 08:30:06

--------------------------------------------------Start of Discovery Import Phase

Data is for KEY1PLEX.KEY1

Discovered 104 address spaces.

Discovery Import Phase complete - 0.851825 seconds--------------------------------------------------Starting PDB Import Phase

Loaded from target PDB:1 Group(s)1 System(s)1 Application Group(s)16 Application(s)

PDB Import Phase Complete - 0.278650 seconds--------------------------------------------------Starting Policy Generation Phase

Generating policy for KEY1PLEX.KEY1

Policy mapping results:91 Address spaces excluded by policy mapping rules.13 Address spaces included.

Figure 58. The SUMMARY Report

Preloader Reports


ARMGROUP ReportThis report contains details of Automatic Restart Management Groups andElements that were discovered. Note that it does not contain any details aboutwhich systems, if any, each element could be moved to.

System Automation for z/OS V3R5 Discovery Import--------------------------------------------------Parameters

SA Mapping file..........: KEY.DRIVER.AINGIMAPInstallation mapping file: KEY.DRIVER.AINGIMAPReport File..............: MIK.DISCO.REPORT.LOAD1

Generated: 24 Feb 2014 08:30:06

XCF Groups and members:

XCF Group: SYSXCFId : GROUP_JES2_XCFMembers:KEY1 KEY1.XCFAS - SYSTEM_XCFASKEY2 KEY2.XCFASKEY3 KEY3.XCFASKEY4 KEY4.XCFAS

XCF Group: SYSGRSId : GROUP_SYSTEM_GRSMembers:KEY1 KEY1.GRS - SYSTEM_GRSKEY2 KEY2.GRSKEY3 KEY3.GRSKEY4 KEY4.GRS

Figure 59. The XCFGROUP Report

Preloader Reports


ASDETAIL ReportThis report contains details of the address spaces discovered, united with their USSdata, Automatic Restart Management data and after the advanced identificationheuristics have been run. This data is the input into the policy mapping phase.



Generated: 10 Mar 2014 07:47:46

Extracting Automatic Restart Management data

Groups and members:

Group : DEFAULTMembers:SYS_RRS_KEY1 NET@IPSVM EZAY1TCPIP M8SGN801001M9SGN901001 N913001 M9DGN90D001 HSAAM_KEY1$$$$1DB2$D911 I911001 OPCKEY1OPCI DBNASNA1OPCKEY1OPC8 DXNAINA1001

Elements:

Element: SYS_RRS_KEY1Jobname: RRSGroup : DEFAULTSystems: KEY1 -> KEY1Level : 2

Element: NET@IPSVMJobname: NETGroup : DEFAULTSystems: KEY1 -> KEY1Level : 1

Element: EZAY1TCPIPJobname: TCPIPGroup : DEFAULTSystems: KEY1 -> KEY1Level : 1

Figure 60. The ARM Group Report

Preloader Reports


EXCLUDE ReportThis report contains details of all address spaces and whether or not they wereincluded in the flat file, along with an identification of the rule in the policymapping file that excluded them (or an indication that they did not match any ofthe rules in the policy mapping files).



Generated: 24 Feb 2014 08:30:06

Address space: - ALLOCAS - 0012Id : SYSTEM_ALLOCASProc : Step : ALLOCASProg : Parms :Subsys : Assoc :User : Type : *Prefix : Job px:Primary: Sched :ARM Ele:ARM Grp:ARM Lvl: Local :USS Data : <-Jobname: User :Command:

Address space: - ANTAS000 - 000CId : SYSTEM_ANTASProc : IEFPROC Step : ANTAS000Prog : ANTXAINI Parms :Subsys : Assoc :User : Type : *Prefix : Job px:

Primary: Sched : MSTRARM Ele:ARM Grp:ARM Lvl: Local :USS Data : <-Jobname: User :Command:

Figure 61. The ASDETAIL Report

Preloader Reports


CONSTRCT ReportThis report contains three distinct sections. The first section describes the loadingof data from the Target PDB Extract. This shows all GRPs, SYSs, APGs and APLsloaded.

The second section describes the processing of the discovered data, seeing whichexisting entries can be reused and deciding if new entries need to be created. Atthe end of this section, the preloader has the PDB delta information in storage.



Generated: 24 Feb 2014 08:30:06

Exclusion report

ALLOCAS Excluded by mapping (23)ANTAS000 Excluded by mapping (23)ANTMAIN Excluded by mapping (23)APPC Excluded by mapping (23)ASCH Excluded by mapping (23)AUXMON Excluded by mapping (23)AXR Excluded by mapping (23)AXR02 Excluded by mapping (23)BPXOINIT Excluded by mapping (23)CATALOG Excluded by mapping (23)CAZ0 Excluded by mapping (23)CEA Excluded by mapping (23)EYUX310 Excluded by mapping (23)FFST Excluded by mapping (23)FTPS1 Excluded by mapping (23)GEOXHSWP Included by mapping (12)GRS Excluded by mapping (23)GRSMON Excluded by mapping (23)HSM Excluded by mapping (23)IEFSCHAS Excluded by mapping (23)

Figure 62. The EXCLUDE Report

Loading data from Target PDB

Entry for sysplex GRP: KEY1PLEXSignature : GRP-KEY1PLEXLoaded GRP: KEY1PLEX (Index: 1)

Entry for system SYS: KEY1PLEX.KEY2Signature : SYS-KEY2Loaded SYS: KEY2 (Index: 1)...linked to GRP KEY1PLEX

Entry for APL BPXOINITSignature : APL-USS-BPXOINIT-0-0Loaded APL: BPXOINIT (Index: 1)

Figure 63. The CONSTRCT Report

Preloader Reports


The third section describes the creation of the flat file to hold the update, includingthe editing of the model policy, the cross referencing of model resources to realresources for relationships and the inclusion of classes from the KBs.

SYMBOLS ReportThis report has three sections. The first section shows the symbols that werediscovered on the target system and their values:

Generating policy for KEY1PLEX.KEY1

Entry for sysplex GRP: KEY1PLEXSignature: GRP-KEY1PLEX...mapped to GRP KEY1PLEX (index: 1)

Entry for system SYS: KEY1PLEX.KEY1Signature : SYS-KEY1Creating New SYS: KEY1 (Index: 2)...linked to GRP KEY1PLEX

Entry for APG LK_KEY1PLEX_KEY1Signature : APG-SYSTEM-BASIC--LK_KEY1PLEX_KEY1Creating New APG: LK_KEY1PLEX_KEY1 (Index: 2)...linked to SYS KEY1

Address space: 001F - GEOXHSWPEntry for APL GEOXHSWPSignature : APL-zOS--GEOXHSWP-HIPER-MSTR-...sig mapped to APL GEOXHSWP (index: 6)...linked to AGP: LK_KEY1PLEX_KEY1

Figure 64. The CONSTRCT Report (second section)

Constructing flat file for: KEY1PLEX.KEY1

NEW SYS KEY1--> GRP KEY1PLEX

NEW APG LK_KEY1PLEX_KEY1--> SYS KEY1

UPD APL GEOXHSWP--> APG LK_KEY1PLEX_KEY1

UPD APL JES2--> APG LK_KEY1PLEX_KEY1

UPD APL LLA--> APG LK_KEY1PLEX_KEY1

Figure 65. The CONSTRCT Report (third section)

Preloader Reports


The second section shows additional variables that were defined in theINGSMVRU and INGSMVRS mapping files:

The third section shows any errors that occurred in processing formulas usingthese symbols.

KBIMPORT ReportThis report first lists the data imported from the SA KB and then the dataimported from the User KB.

Discovered symbols:

&BACKFPT. = IPSFO&BPXPARM. = 10,LO&BPXSHARE. =&CLOCK. = ET&CLOCKEV. =&CNMNETID. = DEIBMIPS&CNMPRTCT. = IPSFN&CNMRODM. = EKGXRODM&CNMTCPN. = TCPIP&COUPLE. = SY&EKGYRODM. = EKGYRODM&FOCALPT. = IPUNM&FSTYPE. = HFS

Figure 66. The SYMBOLS Report

Loading VAR Formulas from: KEY.DRIVER.AINGIMAP(INGSMVRS)

Loaded variables:

Loading VAR Formulas from: KEY.DRIVER.AINGIMAP(INGSMVRU)

Loaded variables:

AOCCLONE1 &SYSCLONE. Y2AOCCLONE2 RIGHT(&SYSNAME.,1) 2AOCCLONE3 RIGHT(&SYSNAME.,2) Y2

Figure 67. The SYMBOLS Report (second section)

Preloader Reports


KBMAP ReportThis report is a compiled 'index' of the SA and User KBs. It lists the entries foundfor each type (APL, APG, and so on) along with an indication of which KB theywere found in.

Loading KB Data from: INGSAQRY as S

Loaded: S GRP SYSPLEX1 - 7 recordsLoaded: S SYS SYS1 - 77 recordsLoaded: S SYS SYS2 - 77 recordsLoaded: S SYS SYS3 - 77 recordsLoaded: S APG AM_X - 23 recordsLoaded: S APG BASE_APPL - 23 recordsLoaded: S APG BASE_SUPP - 23 recordsLoaded: S APG BASE_SYS - 23 recordsLoaded: S APG BASE_USS - 46 recordsLoaded: S APG BBO_ADMIN - 23 recordsLoaded: S APG BBO_CELL - 23 recordsLoaded: S APG BBO_DMGR - 23 recordsLoaded: S APG BBO_DMN - 23 records...Loaded: S APL ASCH - 64 recordsLoaded: S APL BBO_CLASS - 79 recordsLoaded: S APL BLSJPRMI - 52 recordsLoaded: S APL B1AGT - 74 recordsLoaded: S APL B1DMGR - 74 recordsLoaded: S APL B1DMN - 98 recordsLoaded: S APL B1SR1 - 74 recordsLoaded: S APL C_AM - 106 recordsLoaded: S APL C_APPL - 73 recordsLoaded: S APL C_CICS - 298 recordsLoaded: S APL C_CICS_CMAS - 81 recordsLoaded: S APL C_CICS_SHAREDSERVERS - 82 recordsLoaded: S APL C_CICS_TG - 79 records...

Figure 68. The KBIMPORT Report

Preloader Reports


SUBSYS ReportThis report gives a list of all of the z/OS subsystems that were discovered, andidentifies the address space providing the subsystem, any other address spaces thatare associated with it and any address spaces that are scheduled under it.

Entries for type: APL

S U APL AM SA Automation ManagerS APL AM2 Spare SA Automation ManagerS U APL APPC Advanced Peer-to-Peer CommunicationS U APL ASCH APPC SchedulerS APL BBO_CLASS WebSphere class with general definitionsS APL BLSJPRMI Build SNAP Tables for IPCSS APL B1AGT WAS V7 Node AgentS APL B1DMGR WAS V7 Deployment ManagerS APL B1DMN WAS V7 DaemonS APL B1SR1 WAS V7 Application ServerS APL C_AM Class for Automation Manager DefinitionsS APL C_APPL Class for general APL definitionsS APL C_CICS Class for CICSS APL C_CICS_CMAS Class for CICS CPSMCMAS...S APL TWSTRKR TWS/OPC TrackerS U APL VLF Virtual Lookaside FacilityS U APL VTAM Virtual Telecommunication Access MethodS APL WEBSRV IBM HTTP ServerS APL ZFS z/OS File System

U APL AADUMMY Used for checking policy defsU APL ABC XPCS TestU APL ABC2 XPCS TestU APL AM1 Automation manager...U APL BOBHTST1 asU APL BOBHTST2 test scheduler workitems

Figure 69. The KBMAP Report

Subsystem : JES2Provider : JES2 - SYSTEM_JESSchedules : EYUCAS1B GRSMON IMS921I1 IMS921OM IMS921SI IMS922I1IMS922OM IMS922SI IMS923I1 JES2AUX MIKDISC2 NETNETBTST2 OAM OPCH OPCI OPC82S OPC85SPCAUTH PORTMAP Q3E1MSTR RASP SDSF SMSSMSPDSE SNA2DBM1 SNA2DIST SNA2IRLM SNA2MSTR SYSLOGD1TCPIP TNF TN3270 TN3270F TN3270N TRACETSO VMCF YWHITAM2

Subsystem : MSTRProvider : -Schedules : ANTAS000 ANTMAIN APPC ASCH AXR BPXOINITCATALOG CEA DEVMAN DFSZFS DUMPSRV FFSTGEOXHSWP IOSAS JESXCF JES2 JES2MON LLAMVSNFSCS NETBSSI5 OMVS RACF RESOLVER RRSSMF VLF WLM XCFAS

Subsystem : SNA2Provider : SNA2MSTR - DB2_MSTRAssociated: SNA2DBM1 SNA2DIST

Figure 70. The SUBSYS Report

Preloader Reports


Appendix G. Notices

This information was developed for products and services offered in the US. Thismaterial might be available from IBM in other languages. However, you may berequired to own a copy of the product or product version in that language in orderto access it.

IBM may not offer the products, services, or features discussed in this document inother countries. Consult your local IBM representative for information on theproducts and services currently available in your area. Any reference to an IBMproduct, program, or service is not intended to state or imply that only that IBMproduct, program, or service may be used. Any functionally equivalent product,program, or service that does not infringe any IBM intellectual property right maybe used instead. However, it is the user's responsibility to evaluate and verify theoperation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matterdescribed in this document. The furnishing of this document does not grant youany license to these patents. You can send license inquiries, in writing, to:

IBM Director of LicensingIBM CorporationNorth Castle Drive, MD-NC119Armonk, NY 10504-1785US

For license inquiries regarding double-byte character set (DBCS) information,contact the IBM Intellectual Property Department in your country or sendinquiries, in writing, to:

Intellectual Property LicensingLegal and Intellectual Property LawIBM Japan Ltd.19-21, Nihonbashi-Hakozakicho, Chuo-kuTokyo 103-8510, Japan

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THISPUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHEREXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIEDWARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESSFOR A PARTICULAR PURPOSE. Some jurisdictions do not allow disclaimer ofexpress or implied warranties in certain transactions, therefore, this statement maynot apply to you.

This information could include technical inaccuracies or typographical errors.Changes are periodically made to the information herein; these changes will beincorporated in new editions of the publication. IBM may make improvementsand/or changes in the product(s) and/or the program(s) described in thispublication at any time without notice.

Any references in this information to non-IBM websites are provided forconvenience only and do not in any manner serve as an endorsement of those


websites. The materials at those websites are not part of the materials for this IBMproduct and use of those websites is at your own risk.

IBM may use or distribute any of the information you provide in any way itbelieves appropriate without incurring any obligation to you.

Licensees of this program who wish to have information about it for the purposeof enabling: (i) the exchange of information between independently createdprograms and other programs (including this one) and (ii) the mutual use of theinformation which has been exchanged, should contact:

IBM Director of LicensingIBM CorporationNorth Castle Drive, MD-NC119Armonk, NY 10504-1785US

Such information may be available, subject to appropriate terms and conditions,including in some cases, payment of a fee.

The licensed program described in this document and all licensed materialavailable for it are provided by IBM under terms of the IBM Customer Agreement,IBM International Program License Agreement or any equivalent agreementbetween us.

The performance data and client examples cited are presented for illustrativepurposes only. Actual performance results may vary depending on specificconfigurations and operating conditions.

Information concerning non-IBM products was obtained from the suppliers ofthose products, their published announcements or other publicly available sources.IBM has not tested those products and cannot confirm the accuracy ofperformance, compatibility or any other claims related to non-IBM products.Questions on the capabilities of non-IBM products should be addressed to thesuppliers of those products.

Statements regarding IBM's future direction or intent are subject to change orwithdrawal without notice, and represent goals and objectives only.

All IBM prices shown are IBM's suggested retail prices, are current and are subjectto change without notice. Dealer prices may vary.

This information is for planning purposes only. The information herein is subject tochange before the products described become available.

This information contains examples of data and reports used in daily businessoperations. To illustrate them as completely as possible, the examples include thenames of individuals, companies, brands, and products. All of these names arefictitious and any similarity to actual people or business enterprises is entirelycoincidental.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, whichillustrate programming techniques on various operating platforms. You may copy,modify, and distribute these sample programs in any form without payment to


IBM, for the purposes of developing, using, marketing or distributing applicationprograms conforming to the application programming interface for the operatingplatform for which the sample programs are written. These examples have notbeen thoroughly tested under all conditions. IBM, therefore, cannot guarantee orimply reliability, serviceability, or function of these programs. The sampleprograms are provided "AS IS", without warranty of any kind. IBM shall not beliable for any damages arising out of your use of the sample programs.

Each copy or any portion of these sample programs or any derivative work must include a copyright notice as follows:© (your company name) (year).Portions of this code are derived from IBM Corp. Sample Programs.© Copyright IBM Corp. _enter the year or years_.

TrademarksIBM, the IBM logo, and ibm.com are trademarks or registered trademarks ofInternational Business Machines Corp., registered in many jurisdictions worldwide.Other product and service names might be trademarks of IBM or other companies.A current list of IBM trademarks is available on the web at "Copyright andtrademark information" at www.ibm.com/legal/copytrade.shtml.

Terms and conditions for product documentationPermissions for the use of these publications are granted subject to the followingterms and conditions.

Applicability

These terms and conditions are in addition to any terms of use for the IBMwebsite.

Personal use

You may reproduce these publications for your personal, noncommercial useprovided that all proprietary notices are preserved. You may not distribute, displayor make derivative work of these publications, or any portion thereof, without theexpress consent of IBM.

Commercial use

You may reproduce, distribute and display these publications solely within yourenterprise provided that all proprietary notices are preserved. You may not makederivative works of these publications, or reproduce, distribute or display thesepublications or any portion thereof outside your enterprise, without the expressconsent of IBM.

Rights

Except as expressly granted in this permission, no other permissions, licenses orrights are granted, either express or implied, to the publications or anyinformation, data, software or other intellectual property contained therein.

IBM reserves the right to withdraw the permissions granted herein whenever, in itsdiscretion, the use of the publications is detrimental to its interest or, asdetermined by IBM, the above instructions are not being properly followed.

Appendix G. Notices 313

http://www.ibm.com/legal/us/en/copytrade.shtml

You may not download, export or re-export this information except in fullcompliance with all applicable laws and regulations, including all United Statesexport laws and regulations.

IBM MAKES NO GUARANTEE ABOUT THE CONTENT OF THESEPUBLICATIONS. THE PUBLICATIONS ARE PROVIDED "AS-IS" AND WITHOUTWARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDINGBUT NOT LIMITED TO IMPLIED WARRANTIES OF MERCHANTABILITY,NON-INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE.


Glossary

This glossary includes terms and definitions from:v The IBM Dictionary of Computing New York:

McGraw-Hill, 1994.v The American National Standard Dictionary for

Information Systems, ANSI X3.172-1990,copyright 1990 by the American NationalStandards Institute (ANSI). Copies can bepurchased from the American NationalStandards Institute, 1430 Broadway, New York,New York 10018. Definitions are identified bythe symbol (A) after the definition.

v The Information Technology Vocabulary developedby Subcommittee 1, Joint Technical Committee1, of the International Organization forStandardization and the InternationalElectrotechnical Commission (ISO/IECJTC1/SC1). Definitions of published parts ofthis vocabulary are identified by the symbol (I)after the definition; definitions taken from draftinternational standards, committee drafts, andworking papers being developed by ISO/IECJTC1/SC1 are identified by the symbol (T) afterthe definition, indicating that final agreementhas not yet been reached among theparticipating National Bodies of SC1.

The following cross-references are used in thisglossary:

Contrast with. This refers to a term that hasan opposed or substantively differentmeaning.Deprecated term for. This indicates that theterm should not be used. It refers to apreferred term, which is defined in its properplace in the glossary.See. This refers the reader to multiple-wordterms in which this term appears.See also. This refers the reader to terms thathave a related, but not synonymous, meaning.Synonym for. This indicates that the term hasthe same meaning as a preferred term, whichis defined in the glossary.Synonymous with. This is a backwardreference from a defined term to all otherterms that have the same meaning.

A

ACF See automation configuration file.

ACF/NCPAdvanced Communications Function for

the Network Control Program. SeeAdvanced Communications Function andNetwork Control Program.

ACF/VTAMAdvanced Communications Function forthe Virtual Telecommunications AccessMethod. Synonym for VTAM. SeeAdvanced Communications Function andVirtual Telecommunications AccessMethod.

active monitoringIn SA z/OSautomation control file, theacquiring of resource status informationby soliciting such information at regular,user-defined intervals. See also passivemonitoring.

adapterHardware card that enables a device, suchas a workstation, to communicate withanother device, such as a monitor, aprinter, or some other I/O device.

adjacent hostsSystems connected in a peer relationshipusing adjacent NetView sessions forpurposes of monitoring and control.

adjacent NetViewIn SA z/OS, the system defined as thecommunication path between twoSA z/OS systems that do not have adirect link. An adjacent NetView is usedfor message forwarding and as acommunication link between twoSA z/OS systems. For example, theadjacent NetView is used when sendingresponses from a focal point to a remotesystem.

Advanced Communications Function (ACF)A group of IBM licensed programs(principally VTAM, TCAM, NCP, andSSP) that use the concepts of SystemsNetwork Architecture (SNA), includingdistribution of function and resourcesharing.

advanced program-to-program communication(APPC)

A set of inter-program communicationservices that support cooperativetransaction processing in a Systems


Network Architecture (SNA) network.APPC is the implementation, on a givensystem, of SNA's logical unit type 6.2.

Advanced Workload Analysis Reporter (zAware)IBM analytics appliance running in a zSystems partition, activated in zACImode. Customers can use the appliance tomonitor the console message streams ofother LPARs running in the same Systemz cluster and create trend reports.Exploiting zAware and these trend reportscan help to better predict OS outages orperformance degradations and initiateproactive clusters.

alert In SNA, a record sent to a systemproblem management focal point or to acollection point to communicate theexistence of an alert condition.

In NetView, a high-priority event thatwarrants immediate attention. A databaserecord is generated for certain event typesthat are defined by user-constructedfilters.

alert conditionA problem or impending problem forwhich some or all of the process ofproblem determination, diagnosis, andresolution is expected to require action ata control point.

alert focal-point systemSee NPDA focal point system.

alert thresholdAn application or volume service valuethat determines the level at whichSA z/OS changes the associated icon inthe graphical interface to the alert color.SA z/OS may also issue an alert. Seewarning threshold.

AMC See Automation Manager Configuration.

The Auto Msg Classes entry type.

American Standard Code for InformationInterchange (ASCII)

A standard code used for informationexchange among data processing systems,data communication systems, andassociated equipment. ASCII uses a codedcharacter set consisting of 7-bit codedcharacters (8-bit including parity check).The ASCII set consists of control

characters and graphic characters. See alsoExtended Binary Coded DecimalInterchange Code.

APF See authorized program facility.

API See application programming interface.

APPC See advanced program-to-programcommunication.

applicationIn SA z/OS, applications refer to z/OSsubsystems, started tasks, or jobs that areautomated and monitored by SA z/OS.On SNMP-capable processors, applicationcan be used to refer to a subsystem orprocess.

Application entryA construct, created with thecustomization dialogs, used to representand contain policy for an application.

application groupA named set of applications. Anapplication group is part of an SA z/OSenterprise definition and is used formonitoring purposes.

application programA program written for or by a user thatapplies to the user's work, such as aprogram that does inventory or payroll.

A program used to connect andcommunicate with stations in a network,enabling users to performapplication-oriented activities.

application programming interface (API)An interface that allows an applicationprogram that is written in a high-levellanguage to use specific data or functionsof the operating system or anotherprogram.

ApplicationGroup entryA construct, created with thecustomization dialogs, used to representand contain policy for an applicationgroup.

ARM See automatic restart management.

ASCB Address space control block.

ASCB statusAn application status derived bySA z/OS running a routine (the ASCBchecker) that searches the z/OS addressspace control blocks (ASCBs) for address


spaces with a particular job name. The jobname used by the ASCB checker is the jobname defined in the customization dialogfor the application.

ASCII See American Standard Code forInformation Interchange.

ASF See automation status file.

authorized program facility (APF)A facility that permits identification ofprograms that are authorized to userestricted functions.

automated console operations (ACO)The use of an automated procedure toreplace or simplify the action that anoperator takes from a console in responseto system or network events.

automated functionSA z/OS automated functions areautomation operators, NetView autotasksthat are assigned to perform specificautomation functions. However, SA z/OSdefines its own synonyms, or automatedfunction names, for the NetView autotasks,and these function names are referred toin the sample policy databases providedby SA z/OS. For example, theautomation operator AUTBASEcorresponds to the SA z/OS automatedfunction BASEOPER.

automatic restart management (ARM)A z/OS recovery function that improvesthe availability of specified subsystemsand applications by automaticallyrestarting them under certaincircumstances. Automatic restartmanagement is a function of theCross-System Coupling Facility (XCF)component of z/OS.

automatic restart management element nameIn MVS 5.2 or later, z/OS automaticrestart management requires thespecification of a unique sixteen charactername for each address space that registerswith it. All automatic restart managementpolicy is defined in terms of the elementname, including the SA z/OS interfacewith it.

automationThe automatic initiation of actions inresponse to detected conditions or events.SA z/OS provides automation for z/OSapplications, z/OS components, and

remote systems that run z/OS. SA z/OSalso provides tools that can be used todevelop additional automation.

automation agentIn SA z/OS, the automation function issplit up between the automation managerand the automation agents. Theobserving, reacting and doing parts arelocated within the NetView address space,and are known as the automation agents.The automation agents are responsiblefor:v Recovery processingv Message processingv Active monitoring: they propagate

status changes to the automationmanager

automation configuration fileThe SA z/OS customization dialogs mustbe used to build the automationconfiguration file. It consists of:v The automation manager configuration

file (AMC)v The NetView automation table (AT)v The NetView message revision table

(MRT)v The MPFLSTxx member

automation control file (ACF)In SA z/OS, a file that containssystem-level automation policyinformation. There is one masterautomation control file for each NetViewsystem that SA z/OS is installed on.Additional policy information and allresource status information is containedin the policy database (PDB). TheSA z/OS customization dialogs must beused to build the automation control files.They must not be edited manually.

automation flagsIn SA z/OS, the automation policysettings that determine the operatorfunctions that are automated for aresource and the times during whichautomation is active. When SA z/OS isrunning, automation is controlled byautomation flag policy settings andoverride settings (if any) entered by theoperator. Automation flags are set usingthe customization dialogs.

automation managerIn SA z/OS, the automation function issplit up between the automation manager

Glossary 317

and the automation agents. Thecoordination, decision making andcontrolling functions are processed byeach sysplex's automation manager.

The automation manager contains amodel of all of the automated resourceswithin the sysplex. The automation agentsfeed the automation manager with statusinformation and perform the actions thatthe automation manager tells them to.

The automation manager providessysplex-wide automation.

Automation Manager ConfigurationThe Automation Manager Configurationfile (AMC) contains an image of theautomated systems in a sysplex or of astandalone system. See also automationconfiguration file.

Automation NetViewIn SA z/OS the NetView that performsroutine operator tasks with commandprocedures or uses other ways ofautomating system and networkmanagement, issuing automatic responsesto messages and management servicesunits.

automation operatorNetView automation operators areNetView autotasks that are assigned toperform specific automation functions.See also automated function. NetViewautomation operators may receivemessages and process automationprocedures. There are no logged-on usersassociated with automation operators.Each automation operator is an operatingsystem task and runs concurrently withother NetView tasks. An automationoperator could be set up to handle JES2messages that schedule automationprocedures, and an automation statementcould route such messages to theautomation operator. Similar to operatorstation task. SA z/OS message monitortasks and target control tasks areautomation operators.

automation policyThe policy information governingautomation for individual systems. Thisincludes automation for applications,z/OS subsystems, z/OS data sets, andz/OS components.

automation policy settingsThe automation policy informationcontained in the automation control file.This information is entered using thecustomization dialogs. You can display ormodify these settings using thecustomization dialogs.

automation procedureA sequence of commands, packaged as aNetView command list or a commandprocessor written in a high-levellanguage. An automation procedureperforms automation functions and runsunder NetView.

automation routinesIn SA z/OS, a set of self-containedautomation routines that can be calledfrom the NetView automation table, orfrom user-written automation procedures.

automation status file (ASF)In SA z/OS, a file containing statusinformation for each automatedsubsystem, component or data set. Thisinformation is used by SA z/OSautomation when taking action or whendetermining what action to take. InRelease 2 and above of AOC/MVS, statusinformation is also maintained in theoperational information base.

automation table (AT)See NetView automation table.

autotaskA NetView automation task that receivesmessages and processes automationprocedures. There are no logged-on usersassociated with autotasks. Each autotaskis an operating system task and runsconcurrently with other NetView tasks.An autotask could be set up to handleJES2 messages that schedule automationprocedures, and an automation statementcould route such messages to theautotasks. Similar to operator station task.SA z/OS message monitor tasks andtarget control tasks are autotasks. Alsocalled automation operator.

availableIn VTAM programs, pertaining to alogical unit that is active, connected,enabled, and not at its session limit.

B


Base Control Program (BCP)A program that provides essential servicesfor the MVS and z/OS operating systems.The program includes functions thatmanage system resources. These functionsinclude input/output, dispatch units ofwork, and the z/OS UNIX SystemServices kernel. See also Multiple VirtualStorage and z/OS.

basic modeA central processor mode that does notuse logical partitioning. Contrast withlogically partitioned mode.

BCP See Base Control Program.

BCP Internal InterfaceProcessor function of System z processorfamilies. It allows for communicationbetween basic control programs such asz/OS and the processor support elementin order to exchange information or toperform processor control functions.Programs using this function can performhardware operations such as ACTIVATEor SYSTEM RESET.

beaconingThe repeated transmission of a frame ormessages (beacon) by a console orworkstation upon detection of a linebreak or outage.

blade A hardware unit that providesapplication-specific services andcomponents. The consistent size andshape (or form factor) of each bladeallows it to fit in a BladeCenter chassis.

BladeCenter chassisA modular chassis that can containmultiple blades, allowing the individualblades to share resources such asmanagement, switch, power, and blowermodules.

BookManager®

An IBM product that lets users viewsoftcopy documents on theirworkstations.

C

central processor (CP)The part of the computer that contains thesequencing and processing facilities forinstruction execution, initial program load(IPL), and other machine operations.

central processor complex (CPC)A physical collection of hardware thatconsists of central storage, (one or more)central processors, (one or more) timers,and (one or more) channels.

central siteIn a distributed data processing network,the central site is usually defined as thefocal point for alerts, application design,and remote system management taskssuch as problem management.

CFR/CFS and ISC/ISRI/O operations can display and returndata about integrated system channels(ISC) connected to a coupling facility andcoupling facility receiver (CFR) channelsand coupling facility sender (CFS)channels.

channelA path along which signals can be sent;for example, data channel, outputchannel. See also link.

channel path identifierA system-unique value assigned to eachchannel path.

channel-attachedAttached directly by I/O channels to ahost processor (for example, achannel-attached device).

Attached to a controlling unit by cables,rather than by telecommunication lines.Contrast with link-attached. Synonymouswith local.

CHPIDIn SA z/OS, channel path ID; the addressof a channel.

CHPID portA label that describes the system name,logical partitions, and channel paths.

CI See console integration.

CICS/VSCustomer Information Control System forVirtual Storage. See Customer InformationControl System.

CLIST See command list.

clone A set of definitions for applicationinstances that are derived from a basicapplication definition by substituting anumber of different system-specific valuesinto the basic definition.

Glossary 319

clone IDA generic means of handlingsystem-specific values such as the MVSSYSCLONE or the VTAM subareanumber. Clone IDs can be substituted intoapplication definitions and commands tocustomize a basic application definitionfor the system that it is to be instantiatedon.

CNC A channel path that transfers databetween a host system image and anESCON control unit. It can bepoint-to-point or switchable.

commandA request for the performance of anoperation or the execution of a particularprogram.

command facilityThe component of NetView that is a basefor command processors that can monitor,control, automate, and improve theoperation of a network. The successor toNCCF.

command list (CLIST)A list of commands and statements,written in the NetView command listlanguage or the REXX language, designedto perform a specific function for the user.In its simplest form, a command list is alist of commands. More complexcommand lists incorporate variablesubstitution and conditional logic, makingthe command list more like aconventional program. Command lists aretypically interpreted rather than beingcompiled.

In SA z/OS, REXX command lists thatcan be used for automation procedures.

command procedureIn NetView, either a command list or acommand processor.

command processorA module designed to perform a specificfunction. Command processors, which canbe written in assembler or a high-levellanguage (HLL), are issued as commands.

command processor control blockAn I/O operations internal control blockthat contains information about thecommand being processed.

Command Tree/2An OS/2-based program that helps youbuild commands on an OS/2 window,then routes the commands to thedestination you specify (such as a 3270session, a file, a command line, or anapplication program). It provides thecapability for operators to buildcommands and route them to a specifieddestination.

common commandsThe SA z/OS subset of the CPCoperations management commands.

Common User Access (CUA) architectureGuidelines for the dialog between ahuman and a workstation or terminal.

communication controllerA type of communication control unitwhose operations are controlled by one ormore programs stored and executed inthe unit or by a program executed in aprocessor to which the controller isconnected. It manages the details of linecontrol and the routing of data through anetwork.

communication lineDeprecated term for telecommunicationline.

connectivity viewIn SA z/OS, a display that uses graphicimages for I/O devices and lines to showhow they are connected.

console automationThe process of having NetView facilitiesprovide the console input usually handledby the operator.

console connectionIn SA z/OS, the 3270 or ASCII (serial)connection between a PS/2 computer anda target system. Through this connection,the workstation appears (to the targetsystem) to be a console.

console integration (CI)A hardware facility that if supported byan operating system, allows operatingsystem messages to be transferredthrough an internal hardware interface fordisplay on a system console. Conversely,it allows operating system commandsentered at a system console to be


transferred through an internal hardwareinterface to the operating system forprocessing.

consolesWorkstations and 3270-type devices thatmanage your enterprise.

Control unitsHardware units that control I/Ooperations for one or more devices. Youcan view information about control unitsthrough I/O operations, and can start orstop data going to them by blocking andunblocking ports.

controllerA unit that controls I/O operations forone or more devices.

converted mode (CVC)A channel operating in converted (CVC)mode transfers data in blocks and a CBYchannel path transfers data in bytes.Converted CVC or CBY channel paths cancommunicate with a parallel control unit.This resembles a point-to-point parallelpath and dedicated connection, regardlesswhether the path passes through a switch.

couple data setA data set that is created through the XCFcouple data set format utility and,depending on its designated type, isshared by some or all of the z/OSsystems in a sysplex. See also sysplexcouple data setand XCF couple data set.

coupling facilityThe hardware element that provideshigh-speed caching, list processing, andlocking functions in a sysplex.

CP See central processor.

CPC See central processor complex.

CPC operations management commandsA set of commands and responses forcontrolling the operation of System/390®

CPCs.

CPC subsetAll or part of a CPC. It contains theminimum resource to support a singlecontrol program.

CPCB See command processor control block.

CPU Central processing unit. Deprecated termfor processor.

cross-system coupling facility (XCF)A component of z/OS that providesfunctions to support cooperation betweenauthorized programs running within asysplex.

CTC The channel-to-channel (CTC) channel cancommunicate with a CTC on another hostfor intersystem communication.

Customer Information Control System (CICS)A general-purpose transactional programthat controls online communicationbetween terminal users and a database fora large number of end users on areal-time basis.

customization dialogsThe customization dialogs are an ISPFapplication. They are used to customizethe enterprise policy, like, for example,the enterprise resources and therelationships between resources, or theautomation policy for systems in theenterprise. How to use these dialogs isdescribed in IBM Tivoli System Automationfor z/OS Customizing and Programming.

CVC See converted mode.

D

DataPower X150zSee IBM Websphere DataPowerIntegration Appliance X150 forzEnterprise (DataPower X150z).

DASDSee direct access storage device.

data services task (DST)The NetView subtask that gathers,records, and manages data in a VSAM fileor a network device that contains networkmanagement information.

data setThe major unit of data storage andretrieval, consisting of a collection of datain one of several prescribed arrangementsand described by control information towhich the system has access.

data set membersMembers of partitioned data sets that areindividually named elements of a largerfile that can be retrieved by name.

DBCS See double-byte character set.

DCCF See disabled console communicationfacility.

Glossary 321

DCF See Document Composition Facility.

DELAY ReportAn RMF report that shows the activity ofeach job in the system and the hardwareand software resources that are delayingeach job.

device A piece of equipment. Devices can beworkstations, printers, disk drives, tapeunits, remote systems or communicationscontrollers. You can see information aboutall devices attached to a particular switch,and control paths and jobs to devices.

DEVR ReportAn RMF report that presents informationabout the activity of I/O devices that aredelaying jobs.

dialog Interactive 3270 panels.

direct access storage device (DASD)A device that allows storage to be directlyaccessed, such as a disk drive.

disabled console communication facility (DCCF)A z/OS component that provideslimited-function console communicationduring system recovery situations.

disk operating system (DOS)An operating system for computersystems that use disks and diskettes forauxiliary storage of programs and data.

Software for a personal computer thatcontrols the processing of programs. Forthe IBM Personal Computer, the full nameis Personal Computer Disk OperatingSystem (PCDOS).

displayTo present information for viewing,usually on the screen of a workstation oron a hardcopy device.

Deprecated term for panel.

distribution managerThe component of the NetView programthat enables the host system to use, send,and delete files and programs in anetwork of computers.

Document Composition Facility (DCF)An IBM licensed program used to formatinput to a printer.

domainAn access method and its application

programs, communication controllers,connecting lines, modems, and attachedworkstations.

In SNA, a system services control point(SSCP) and the physical units (PUs),logical units (LUs), links, link stations,and associated resources that the SSCPcan control with activation requests anddeactivation requests.

double-byte character set (DBCS)A character set, such as Kanji, in whicheach character is represented by a 2-bytecode.

DP enterpriseData processing enterprise.

DSIPARMThis file is a collection of members for thecustomization of NetView.

DST Data Services Task.

E

EBCDICSee Extended Binary Coded DecimalInterchange Code.

ECB See event control block.

EMCS Extended multiple console support. Seealso multiple console support.

ensembleA collection of one or more zEnterprisenodes (including any attached zBX) thatare managed as a single logicalvirtualized system by the UnifiedResource Manager, through the HardwareManagement Console.

ensemble memberA zEnterprise node that has been addedto an ensemble.

enterpriseThe composite of all operational entities,functions, and resources that form thetotal business concern and that require aninformation system.

enterprise monitoringEnterprise monitoring is used bySA z/OS to update the NetViewManagement Console (NMC) resourcestatus information that is stored in theResource Object Data Manager (RODM).Resource status information is acquiredby enterprise monitoring of the Resource


Measurement Facility (RMF) Monitor IIIservice information at user-definedintervals. SA z/OS stores this informationin its operational information base, whereit is used to update the informationpresented to the operator in graphicdisplays.

Enterprise Systems Architecture (ESA)A hardware architecture that reduces theeffort required for managing data sets andextends addressability for system,subsystem, and application functions.

entriesResources, such as processors, entered onpanels.

entry typeResources, such as processors orapplications, used for automation andmonitoring.

environmentData processing enterprise.

error thresholdAn automation policy setting thatspecifies when SA z/OS should stoptrying to restart or recover an application,subsystem or component, or offload adata set.

ESA See Enterprise Systems Architecture.

event In NetView, a record indicatingirregularities of operation in physicalelements of a network.

An occurrence of significance to a task;for example, the completion of anasynchronous operation, such as aninput/output operation.

Events are part of a trigger condition,such that if all events of a triggercondition have occurred, a startup orshutdown of an application is performed.

event control block (ECB)A control block used to represent thestatus of an event.

exception conditionAn occurrence on a system that is adeviation from normal operation.SA z/OS monitoring highlights exceptionconditions and allows an SA z/OSenterprise to be managed by exception.

Extended Binary Coded Decimal InterchangeCode (EBCDIC)

A coded character set of 256 8-bitcharacters developed for therepresentation of textual data. See alsoAmerican Standard Code for InformationInterchange.

extended recovery facility (XRF)A facility that minimizes the effect offailures in z/OS, VTAM, the hostprocessor, or high availability applicationsduring sessions between high availabilityapplications and designated terminals.This facility provides an alternatesubsystem to take over sessions from thefailing subsystem.

F

fallback systemSee secondary system.

field A collection of bytes within a record thatare logically related and are processed asa unit.

file manager commandsA set of SA z/OS commands that readdata from or write data to the automationcontrol file or the operational informationbase. These commands are useful in thedevelopment of automation that usesSA z/OS facilities.

focal pointIn NetView, the focal-point domain is thecentral host domain. It is the centralcontrol point for any managementservices element containing control of thenetwork management data.

focal point systemA system that can administer, manage, orcontrol one or more target systems. Thereare a number of different focal pointsystem associated with IBM automationproducts.

NMC focal point system. The NMC focalpoint system is a NetView system with anattached workstation server and LAN thatgathers information about the state of thenetwork. This focal point system usesRODM to store the data it collects in thedata model. The information stored inRODM can be accessed from anyLAN-connected workstation withNetView Management Console installed.

Glossary 323

NPDA focal point system. This is aNetView system that collects all theNPDA alerts that are generated withinyour enterprise. It is supported byNetView. If you have SA z/OS installedthe NPDA focal point system must be thesame as your NMC focal point system.The NPDA focal point system is alsoknown as the alert focal point system.

SA z/OS Processor Operations focalpoint system. This is a NetView systemthat has SA z/OS host code installed. TheSA z/OS Processor Operations focal pointsystem receives messages from thesystems and operator consoles of themachines that it controls. It provides fullsystems and operations console functionfor its target systems. It can be used toIPL these systems. Note that somerestrictions apply to the HardwareManagement Console for an S/390®

microprocessor cluster.

SA z/OS SDF focal point system. TheSA z/OS SDF focal point system is anSA z/OS NetView system that collectsstatus information from other SA z/OSNetViews within your enterprise.

Status focal point system. In NetView,the system to which STATMON, VTAMand NLDM send status information onnetwork resources. If you have a NMCfocal point, it must be on the same systemas the Status focal point.

Hardware Management Console.Although not listed as a focal point, theHardware Management Console acts as afocal point for the console functions of anS/390 microprocessor cluster. Unlike allthe other focal points in this definition,the Hardware Management Console runson a LAN-connected workstation,

frame For a System/390 microprocessor cluster,a frame contains one or two centralprocessor complexes (CPCs), supportelements, and AC power distribution.

full-screen modeIn NetView, a form of panel presentationthat makes it possible to display thecontents of an entire workstation screen atonce. Full-screen mode can be used forfill-in-the-blanks prompting. Contrastwith line mode.

G

gateway sessionAn NetView-NetView Task session withanother system in which the SA z/OSoutbound gateway operator logs onto theother NetView session without humanoperator intervention. Each end of agateway session has both an inbound andoutbound gateway operator.

generic alertEncoded alert information that uses codepoints (defined by IBM and possiblycustomized by users or applicationprograms) stored at an alert receiver, suchas NetView.

group A collection of target systems definedthrough configuration dialogs. Aninstallation might set up a group to referto a physical site or an organizational orapplication entity.

group entryA construct, created with thecustomization dialogs, used to representand contain policy for a group.

group entry typeA collection of target systems definedthrough the customization dialog. Aninstallation might set up a group to referto a physical site or an organizationalentity. Groups can, for example, be oftype STANDARD or SYSPLEX.

H

Hardware Management Console (HMC)A user interface through which datacenter personnel configure, control,monitor, and manage System z hardwareand software resources. The HMCcommunicates with each central processorcomplex (CPC) through the SupportElement. On an IBM zEnterprise 196(z196), using the Unified ResourceManager on the HMCs or SupportElements, personnel can also create andmanage an ensemble.

Hardware Management Console Application(HWMCA)

A direct-manipulation object-orientedgraphical user interface that provides asingle point of control and single systemimage for hardware elements. TheHWMCA provides grouping support,aggregated and real-time system status


using colors, consolidated hardwaremessages support, consolidated operatingsystem messages support, consolidatedservice support, and hardware commandstargeted at a single system, multiplesystems, or a group of systems.

heartbeatIn SA z/OS, a function that monitors thevalidity of the status forwarding pathbetween remote systems and the NMCfocal point, and monitors the availabilityof remote z/OS systems, to ensure thatstatus information displayed on theSA z/OS workstation is current.

help panelAn online panel that tells you how to usea command or another aspect of aproduct.

hierarchyIn the NetView program, the resourcetypes, display types, and data types thatmake up the organization, or levels, in anetwork.

high-level language (HLL)A programming language that providessome level of abstraction from assemblerlanguage and independence from aparticular type of machine. For theNetView program, the high-levellanguages are PL/I and C.

HLL See high-level language.

host (primary processor)The processor that you enter a commandat (also known as the issuing processor).

host systemIn a coupled system or distributed systemenvironment, the system on which thefacilities for centralized automation run.SA z/OS publications refer to targetsystems or focal-point systems instead ofhosts.

HWMCASee Hardware Management ConsoleApplication.

HypervisorA program that allows multiple instancesof operating systems or virtual servers torun simultaneously on the same hardwaredevice. A hypervisor can run directly onthe hardware, can run within anoperating system, or can be imbedded in

platform firmware. Examples ofhypervisors include PR/SM, z/VM, andPowerVM Enterprise Edition.

I

IBM bladeA customer-acquired, customer-installedselect blade to be managed by IBMzEnterprise Unified Resource Manager.One example of an IBM blade is aPOWER7 blade.

IBM Smart Analyzer for DB2 for z/OSAn optimizer that processes certain typesof data warehouse queries for DB2 forz/OS.

IBM System z Application Assist Processor(zAAP)

A specialized processor that provides aJava execution environment, whichenables Java-based web applications to beintegrated with core z/OS businessapplications and backend databasesystems.

IBM System z Integrated Information Processor(zIIP) A specialized processor that provides

computing capacity for selected data andtransaction processing workloads and forselected network encryption workloads.

IBM Websphere DataPower IntegrationAppliance X150 for zEnterprise (DataPowerX150z) A purpose-built appliance that simplifies,

helps secure, and optimizes XML andWeb services processing.

IBM Enterprise 196 (z196)The newest generation of System z familyof servers built on a new processor chip,with enhanced memory function andcapacity, security, and on demandenhancements to support existingmainframe workloads and large scaleconsolidation.

IBM zEnterprise BladeCenter Extension (zBX)A heterogeneous hardware infrastructurethat consists of a BladeCenter chassisattached to an IBM zEnterprise 196 (z196).A BladeCenter chassis can contain IBMblades or optimizers.

IBM zEnterprise BladeCenter Extension (zBX)blade Generic name for all blade types

supported in an IBM zEnterpriseBladeCenter Extension (zBX). This termincludes IBM blades and optimizers.

Glossary 325

IBM zEnterprise System (zEnterprise)A heterogeneous hardware infrastructurethat can consist of an IBM zEnterprise 196(z196) and an attached IBM zEnterpriseBladeCenter Extension (zBX) Model 002,managed as a single logical virtualizedsystem by the Unified Resource Manager.

IBM zEnterprise Unified Resource ManagerLicensed Internal Code (LIC), also knownas firmware, that is part of the HardwareManagement Console. The UnifiedResource Manager provides energymonitoring and management,goal-oriented policy management,increased security, virtual networking,and data management for the physicaland logical resources of a given ensemble.

I/O operationsThe part of SA z/OS that provides youwith a single point of logical control formanaging connectivity in your active I/Oconfigurations. I/O operations takes anactive role in detecting unusual conditionsand lets you view and change pathsbetween a processor and an I/O device,using dynamic switching (the ESCONdirector). Also known as I/O Ops.

I/O OpsSee I/O operations.

I/O resource numberCombination of channel path identifier(CHPID), device number, etc. See internaltoken.

imagesA grouping of processors and I/O devicesthat you define. You can define asingle-image mode that allows amultiprocessor system to function as onecentral processor image.

IMS See Information Management System.

IMS/VSSee Information ManagementSystem/Virtual Storage.

inboundIn SA z/OS, messages sent to thefocal-point system from the PC or targetsystem.

inbound gateway operatorThe automation operator that receivesincoming messages, commands, andresponses from the outbound gateway

operator at the sending system. Theinbound gateway operator handlescommunications with other systems usinga gateway session.

Information Management System (IMS)Any of several system environmentsavailable with a database manager andtransaction processing that are capable ofmanaging complex databases andterminal networks.

Information Management System/VirtualStorage (IMS/VS)

A database/data communication(DB/DC) system that can managecomplex databases and networks.Synonymous with InformationManagement System.

INGEIO PROCThe I/O operations default procedurename. It is part of the SYS1.PROCLIB.

initial microprogram loadThe action of loading microprograms intocomputer storage.

initial program load (IPL)The initialization procedure that causes anoperating system to commence operation.

The process by which a configurationimage is loaded into storage at thebeginning of a workday or after a systemmalfunction.

The process of loading system programsand preparing a system to run jobs.

initialize automationSA z/OS-provided automation that issuesthe correct z/OS start command for eachsubsystem when SA z/OS is initialized.The automation ensures that subsystemsare started in the order specified in theautomation control files and thatprerequisite applications are functional.

input/output configuration data set (IOCDS)A configuration definition built by theI/O configuration program (IOCP) andstored on disk files associated with theprocessor controller.

input/output support processor (IOSP)The hardware unit that provides I/Osupport functions for the primary supportprocessor and maintenance supportfunctions for the processor controller.


Interactive System Productivity Facility (ISPF)An IBM licensed program that serves as afull-screen editor and dialog manager.Used for writing application programs, itprovides a means of generating standardscreen panels and interactive dialogsbetween the application programmer andthe terminal user. See also Time SharingOption.

interested operator listThe list of operators who are to receivemessages from a specific target system.

internal tokenA logical token (LTOK); name by which theI/O resource or object is known; stored inIODF.

IOCDSSee input/output configuration data set.

IOSP See input/output support processor..

IPL See initial program load.

ISPF See Interactive System ProductivityFacility.

ISPF consoleYou log on to ISPF from this 3270-typeconsole to use the runtime panels for I/Ooperations and SA z/OS customizationpanels.

issuing hostThe base program that you enter acommand for processing with. Seeprimary host.

J

JCL See job control language.

JES See job entry subsystem.

JES2 An MVS subsystem that receives jobs intothe system, converts them to internalformat, selects them for execution,processes their output, and purges themfrom the system. In an installation withmore than one processor, each JES2processor independently controls its jobinput, scheduling, and output processing.See also job entry subsystem and JES3

JES3 An MVS subsystem that receives jobs intothe system, converts them to internalformat, selects them for execution,processes their output, and purges themfrom the system. In complexes that haveseveral loosely coupled processing units,

the JES3 program manages processors sothat the global processor exercisescentralized control over the localprocessors and distributes jobs to themusing a common job queue. See also jobentry subsystem and JES2.

job A set of data that completely defines aunit of work for a computer. A job usuallyincludes all necessary computerprograms, linkages, files, and instructionsto the operating system.

An address space.

job control language (JCL)A problem-oriented language designed toexpress statements in a job that are usedto identify the job or describe itsrequirements to an operating system.

job entry subsystem (JES)An IBM licensed program that receivesjobs into the system and processes alloutput data that is produced by jobs. InSA z/OS publications, JES refers to JES2or JES3, unless otherwise stated. See alsoJES2 and JES3.

K

Kanji An ideographic character set used inJapanese. See also double-byte characterset.

L

LAN See local area network.

line modeA form of screen presentation in whichthe information is presented a line at atime in the message area of the terminalscreen. Contrast with full-screen mode.

link In SNA, the combination of the linkconnection and the link stations joiningnetwork nodes; for example, aSystem/370 channel and its associatedprotocols, a serial-by-bit connection underthe control of synchronous data linkcontrol (SDLC). See synchronous data linkcontrol.

In SA z/OS, link connection is thephysical medium of transmission.

link-attachedDescribes devices that are physicallyconnected by a telecommunication line.Contrast with channel-attached.

Glossary 327

Linux on System zUNIX-like open source operating systemconceived by Linus Torvalds anddeveloped across the internet.

local Pertaining to a device accessed directlywithout use of a telecommunication line.Synonymous with channel-attached.

local area network (LAN)A network in which a set of devices isconnected for communication. They canbe connected to a larger network. See alsotoken ring.

A network that connects several devicesin a limited area (such as a singlebuilding or campus) and that can beconnected to a larger network.

logical partition (LP)A subset of the processor hardware that isdefined to support an operating system.See also logically partitioned mode.

logical switch number (LSN)Assigned with the switch parameter ofthe CHPID macro of the IOCP.

logical token (LTOK)Resource number of an object in theIODF.

logical unit (LU)In SNA, a port through which an enduser accesses the SNA network and thefunctions provided by system servicescontrol points (SSCPs). An LU cansupport at least two sessions, one with anSSCP and one with another LU, and maybe capable of supporting many sessionswith other LUs. See also physical unit andsystem services control point.

logical unit 6.2 (LU 6.2)A type of logical unit that supportsgeneral communications betweenprograms in a distributed processingenvironment. LU 6.2 is characterized by:v A peer relationship between session

partnersv Efficient use of a session for multiple

transactionsv A comprehensive end-to-end error

processingv A generic application program interface

(API) consisting of structured verbsthat are mapped to a productimplementation

Synonym for advancedprogram-to-program communication.

logically partitioned (LPAR) modeA central processor mode that enables anoperator to allocate system processorhardware resources among several logicalpartitions. Contrast with basic mode.

LOGR The sysplex logger.

LP See logical partition.

LPAR See logically partitioned mode.

LSN See logical switch number.

LU See logical unit.

LU 6.2 See logical unit 6.2.

LU 6.2 sessionA session initiated by VTAM on behalf ofan LU 6.2 application program, or asession initiated by a remote LU in whichthe application program specifies thatVTAM is to control the session by usingthe APPCCMD macro. See logical unit 6.2.

LU-LU sessionIn SNA, a session between two logicalunits (LUs) in an SNA network. Itprovides communication between twoend users, or between an end user and anLU services component.

M

MAT Deprecated term for NetView automationtable.

MCA See Micro Channel architecture.

MCS See multiple console support.

memberA specific function (one or more modulesor routines) of a multisystem applicationthat is defined to XCF and assigned to agroup by the multisystem application. Amember resides on one system in thesysplex and can use XCF services tocommunicate (send and receive data) withother members of the same group.

message automation table (MAT)Deprecated term for NetView automationtable.

message classA number that SA z/OS associates with amessage to control routing of themessage. During automated operations,the classes associated with each message


issued by SA z/OS are compared to theclasses assigned to each notificationoperator. Any operator with a classmatching one of the message’s classesreceives the message.

message forwardingThe SA z/OS process of sendingmessages generated at an SA z/OS targetsystem to the SA z/OS focal-pointsystem.

message groupSeveral messages that are displayedtogether as a unit.

message monitor taskA task that starts and is associated with anumber of communications tasks.Message monitor tasks receive inboundmessages from a communications task,determine the originating target system,and route the messages to the appropriatetarget control tasks.

message processing facility (MPF)A z/OS table that screens all messagessent to the z/OS console. The MPFcompares these messages with acustomer-defined list of messages (basedon this message list, messages areautomated and/or suppressed from z/OSconsole display), and marks messages toautomate or suppress. Messages are thenbroadcast on the subsystem interface(SSI).

message suppressionThe ability to restrict the amount ofmessage traffic displayed on the z/OSconsole.

Micro Channel architectureThe rules that define how subsystems andadapters use the Micro Channel bus in acomputer. The architecture defines theservices that each subsystem can or mustprovide.

microprocessorA processor implemented on one or asmall number of chips.

migrationInstallation of a new version or release ofa program to replace an earlier version orrelease.

MP Multiprocessor.

MPF See message processing facility.

MPFLSTxxThe MPFLST member that is built bySA z/OS.

multi-MVS environmentphysical processing system that is capableof operating more than one MVS image.See also MVS image.

multiple console support (MCS)A feature of MVS that permits selectivemessage routing to multiple consoles.

Multiple Virtual Storage (MVS)An IBM operating system that accessesmultiple address spaces in virtual storage.The predecessor of z/OS.

multiprocessor (MP)A CPC that can be physically partitionedto form two operating processorcomplexes.

multisystem applicationAn application program that has variousfunctions distributed across z/OS imagesin a multisystem environment.

multisystem environmentAn environment in which two or moresystems reside on one or more processors.Or one or more processors cancommunicate with programs on the othersystems.

MVS See Multiple Virtual Storage.

MVS imageA single occurrence of the MVS operatingsystem that has the ability to processwork. See also multi-MVS environmentand single-MVS environment.

MVS/ESAMultiple Virtual Storage/EnterpriseSystems Architecture. See z/OS.

MVS/JES2Multiple Virtual Storage/Job EntrySystem 2. A z/OS subsystem that receivesjobs into the system, converts them to aninternal format, selects them forexecution, processes their output, andpurges them from the system. In aninstallation with more than one processor,each JES2 processor independentlycontrols its job input, scheduling, andoutput processing.

N

NAU See network addressable unit.

Glossary 329

See network accessible unit.

NCCF See Network Communications ControlFacility..

NCP See network control program (generalterm).

See Network Control Program (an IBMlicensed program). Its full name isAdvanced Communications Function forthe Network Control Program.Synonymous with ACF/NCP.

NCP/token ring interconnectionA function used by ACF/NCP to supporttoken ring-attached SNA devices. NTRIalso provides translation from tokenring-attached SNA devices (PUs) toswitched (dial-up) devices.

NetViewAn IBM licensed program used tomonitor a network, manage it, anddiagnose network problems. NetViewconsists of a command facility thatincludes a presentation service, commandprocessors, automation based oncommand lists, and a transactionprocessing structure on which the sessionmonitor, hardware monitor, and terminalaccess facility (TAF) network managementapplications are built.

NetView (NCCF) consoleA 3270-type console for NetViewcommands and runtime panels for systemoperations and processor operations.

NetView automation proceduresA sequence of commands, packaged as aNetView command list or a commandprocessor written in a high-levellanguage. An automation procedureperforms automation functions and runsunder the NetView program.

NetView automation table (AT)A table against which the NetViewprogram compares incoming messages. Amatch with an entry triggers the specifiedresponse. SA z/OS entries in the NetViewautomation table trigger an SA z/OSresponse to target system conditions.Formerly known as the messageautomation table (MAT).

NetView command list languageAn interpretive language unique toNetView that is used to write commandlists.

NetView Graphic Monitor Facility (NGMF)Deprecated term for NetViewManagement Console.

NetView hardware monitorThe component of NetView that helpsidentify network problems, such ashardware, software, and microcode, froma central control point using interactivedisplay techniques. Formerly callednetwork problem determination application.

NetView logThe log that NetView records eventsrelating to NetView and SA z/OSactivities in.

NetView Management Console (NMC)A function of the NetView program thatprovides a graphic, topologicalpresentation of a network that iscontrolled by the NetView program. Itprovides the operator different views of anetwork, multiple levels of graphicaldetail, and dynamic resource status of thenetwork. This function consists of a seriesof graphic windows that allows you tomanage the network interactively.Formerly known as the NetView GraphicMonitor Facility (NGMF).

NetView message tableSee NetView automation table.

NetView paths via logical unit (LU 6.2)A type of network-accessible port (VTAMconnection) that enables end users to gainaccess to SNA network resources andcommunicate with each other. LU 6.2permits communication betweenprocessor operations and the workstation.See logical unit 6.2.

NetView-NetView task (NNT)The task that a cross-domain NetViewoperator session runs under. EachNetView program must have aNetView-NetView task to establish oneNNT session. See also operator stationtask.

NetView-NetView task sessionA session between two NetView programsthat runs under a NetView-NetView task.In SA z/OS, NetView-NetView task


sessions are used for communicationbetween focal point and remote systems.

networkAn interconnected group of nodes.

In data processing, a user applicationnetwork. See SNA network.

network accessible unit (NAU)In SNA networking, any device on thenetwork that has a network address,including a logical unit (LU), physicalunit (PU), control point (CP), or systemservices control point (SSCP). It is theorigin or the destination of informationtransmitted by the path control network.Synonymous with network addressableunit.

network addressable unit (NAU)Synonym for network accessible unit.

Network Communications Control Facility(NCCF)

The operations control facility for thenetwork. NCCF consists of a presentationservice, command processors, automationbased on command lists, and atransaction processing structure on whichthe network management applicationsNLDM and NPDA are built. NCCF is aprecursor to the NetView commandfacility.

Network Control Program (NCP)An IBM licensed program that providescommunication controller support forsingle-domain, multiple-domain, andinterconnected network capability. Its fullname is Advanced CommunicationsFunction for the Network ControlProgram.

network control program (NCP)A program that controls the operation ofa communication controller.

A program used for requests andresponses exchanged between physicalunits in a network for data flow control.

Network Problem Determination Application(NPDA)

An NCCF application that helps youidentify network problems, such ashardware, software, and microcode, froma central control point using interactivedisplay methods. The alert manager for

the network. The precursor of theNetView hardware monitor.

Networking NetViewIn SA z/OS the NetView that performsnetwork management functions, such asmanaging the configuration of a network.In SA z/OS it is common to also routealerts to the Networking NetView.

NGMFDeprecated term for NetViewManagement Console.

NGMF focal-point systemDeprecated term for NMC focal pointsystem.

NIP See nucleus initialization program.

NMC focal point systemSee focal point system

NMC workstationThe NMC workstation is the primary wayto dynamically monitor SA z/OSsystems. From the windows, you seemessages, monitor status, view trends,and react to changes before they causeproblems for end users. You can usemultiple windows to monitor multipleviews of the system.

NNT See NetView-NetView task.

notification messageAn SA z/OS message sent to a humannotification operator to provideinformation about significant automationactions. Notification messages are definedusing the customization dialogs.

notification operatorA NetView console operator who isauthorized to receive SA z/OSnotification messages. Authorization ismade through the customization dialogs.

NPDASee Network Problem DeterminationApplication.

NPDA focal-point systemSee focal point system.

NTRI See NCP/token ring interconnection.

nucleus initialization program (NIP)The program that initializes the residentcontrol program; it allows the operator to

Glossary 331

request last-minute changes to certainoptions specified during systemgeneration.

O

objective valueAn average Workflow or Using value thatSA z/OS can calculate for applicationsfrom past service data. SA z/OS uses theobjective value to calculate warning andalert thresholds when none are explicitlydefined.

OCA In SA z/OS, operator console A, theactive operator console for a targetsystem. Contrast with OCB.

OCB In SA z/OS, operator console B, thebackup operator console for a targetsystem. Contrast with OCA.

OPC/ASee Operations Planning andControl/Advanced.

OPC/ESASee Operations Planning andControl/Enterprise Systems Architecture.

Open Systems Adapter (OSA)I/O operations can display the OpenSystem Adapter (OSA) channel logicaldefinition, physical attachment, andstatus. You can configure an OSA channelon or off.

operating system (OS)Software that controls the execution ofprograms and that may provide servicessuch as resource allocation, scheduling,input/output control, and datamanagement. Although operating systemsare predominantly software, partialhardware implementations are possible.(T)

operationsThe real-time control of a hardwaredevice or software function.

Operations Planning and Control/Advanced(OPC/A)

A set of IBM licensed programs thatautomate, plan, and control batchworkload. OPC/A analyzes system andworkload status and submits jobsaccordingly.

Operations Planning and Control/EnterpriseSystems Architecture (OPC/ESA)

A set of IBM licensed programs thatautomate, plan, and control batchworkload. OPC/ESA analyzes system andworkload status and submits jobsaccordingly. The successor to OPC/A.

operatorA person who keeps a system running.

A person or program responsible formanaging activities controlled by a givenpiece of software such as z/OS, theNetView program, or IMS.

A person who operates a device.

In a language statement, the lexical entitythat indicates the action to be performedon operands.

operator consoleA functional unit containing devices thatare used for communications between acomputer operator and a computer. (T)

A display console used forcommunication between the operator andthe system, used primarily to specifyinformation concerning applicationprograms and I/O operations and tomonitor system operation.

In SA z/OS, a console that displaysoutput from and sends input to theoperating system (z/OS, LINUX, VM,VSE). Also called operating system console.In the SA z/OS operator commands andconfiguration dialogs, OC is used todesignate a target system operatorconsole.

operator station task (OST)The NetView task that establishes andmaintains the online session with thenetwork operator. There is one operatorstation task for each network operatorwho logs on to the NetView program.

operator viewA set of group, system, and resourcedefinitions that are associated together formonitoring purposes. An operator viewappears as a graphic display in thegraphical interface showing the status ofthe defined groups, systems, andresources.

OperatorView entryA construct, created with the


customization dialogs, used to representand contain policy for an operator view.

optimizerA special-purpose hardware componentor appliance that can perform a limitedset of specific functions with optimizedperformance when compared to ageneral-purpose processor. Because of itslimited set of functions, an optimizer is anintegrated part of a processingenvironment, rather than a stand-aloneunit. One example of an optimizer is theIBM Smart Analytics Optimizer for DB2for z/OS.

OS See operating system.

OSA See Open Systems Adapter.

OST See operator station task.

outboundIn SA z/OS, messages or commands fromthe focal-point system to the targetsystem.

outbound gateway operatorThe automation operator that establishesconnections to other systems. Theoutbound gateway operator handlescommunications with other systemsthrough a gateway session. Theautomation operator sends messages,commands, and responses to the inboundgateway operator at the receiving system.

P

page The portion of a panel that is shown on adisplay surface at one time.

To transfer instructions, data, or bothbetween real storage and external page orauxiliary storage.

panel A formatted display of information thatappears on a terminal screen. Panels arefull-screen 3270-type displays with amonospaced font, limited color andgraphics.

By using SA z/OS panels you can seestatus, type commands on a commandline using a keyboard, configure yoursystem, and passthru to other consoles.See also help panel.

In computer graphics, a display imagethat defines the locations andcharacteristics of display fields on adisplay surface. Contrast with screen.

parallel channelsParallel channels operate in either byte(BY) or block (BL) mode. You can changeconnectivity to a parallel channeloperating in block mode.

parameterA variable that is given a constant valuefor a specified application and that mayrepresent an application, for example.

An item in a menu for which the userspecifies a value or for which the systemprovides a value when the menu isinterpreted.

Data passed to a program or procedureby a user or another program, specificallyas an operand in a language statement, asan item in a menu, or as a shared datastructure.

partitionA fixed-size division of storage.

In VSE, a division of the virtual addressarea that is available for programprocessing.

On an IBM Personal Computer fixed disk,one of four possible storage areas ofvariable size; one can be accessed byDOS, and each of the others may beassigned to another operating system.

partitionable CPCA CPC that can be divided into 2independent CPCs. See also physicalpartition, single-image mode, MP, andside.

partitioned data set (PDS)A data set in direct access storage that isdivided into partitions, called members,each of which can contain a program, partof a program, or data.

passive monitoringIn SA z/OS, the receiving of unsolicitedmessages from z/OS systems and theirresources. These messages can promptupdates to resource status displays. Seealso active monitoring

PCE A processor controller. Also known as thesupport processor or service processor insome processor families.

PDB See policy database.

PDS See partitioned data set.

Glossary 333

physical partitionPart of a CPC that operates as a CPC inits own right, with its own copy of theoperating system.

physical unit (PU)In SNA, the component that manages andmonitors the resources (such as attachedlinks and adjacent link stations) of a node,as requested by a system services controlpoint (SSCP) through an SSCP-PU session.An SSCP activates a session with thephysical unit to indirectly manage,through the PU, resources of the nodesuch as attached links.

physically partitioned (PP) configurationA mode of operation that allows amultiprocessor (MP) system to function astwo or more independent CPCs havingseparate power, utilities, and maintenanceboundaries. Contrast with single-imagemode.

PLEXID groupPLEXID group or "extended XCFcommunication group" is a term used inconjunction with a sysplex. The PLEXIDgroup includes System AutomationAgents for a subset of a sysplex or for theentire sysplex. It is used to provide XCFcommunication beyond the SAplexboundaries. For a detailed description,refer to "Defining the Extended XCFCommunication Group" in IBM TivoliSystem Automation for z/OS Planning andInstallation.

POI See program operator interface.

policy The automation and monitoringspecifications for an SA z/OS enterprise.See IBM Tivoli System Automation for z/OSDefining Automation Policy.

policy databaseThe automation definitions (automationpolicy) that the automation programmerspecifies using the customization dialog isstored in the policy database. Also knownas the PDB. See also automation policy.

POR See power-on reset.

port System hardware that the I/O devices areattached to.

In an ESCON switch, a port is anaddressable connection. The switch routesdata through the ports to the channel or

control unit. Each port has a name thatcan be entered into a switch matrix, andyou can use commands to change theswitch configuration.

An access point (for example, a logicalunit) for data entry or exit.

A functional unit of a node that data canenter or leave a data network through.

In data communication, that part of adata processor that is dedicated to asingle data channel for the purpose ofreceiving data from or transmitting datato one or more external, remote devices.

power-on reset (POR)A function that re-initializes all thehardware in a CPC and loads the internalcode that enables the CPC to load andrun an operating system. See initialmicroprogram load.

PP See physical partition.

PPI See program to program interface.

PPT See primary POI task.

PR/SMSee Processor Resource/Systems Manager.

primary hostThe base program that you enter acommand for processing at.

primary POI task (PPT)The NetView subtask that processes allunsolicited messages received from theVTAM program operator interface (POI)and delivers them to the controllingoperator or to the command processor.The PPT also processes the initialcommand specified to execute whenNetView is initialized and timer requestcommands scheduled to execute underthe PPT.

primary systemA system is a primary system for anapplication if the application is normallymeant to be running there. SA z/OSstarts the application on all the primarysystems defined for it.

problem determinationThe process of determining the source ofa problem; for example, a programcomponent, machine failure,telecommunication facilities, user orcontractor-installed programs or


|||||||||||||

equipment, environment failure such as apower loss, or user error.

processorA device for processing data fromprogrammed instructions. It may be partof another unit.

In a computer, the part that interprets andexecutes instructions. Two typicalcomponents of a processor are a controlunit and an arithmetic logic unit.

processor controllerHardware that provides support anddiagnostic functions for the centralprocessors.

processor operationsThe part of SA z/OS that monitors andcontrols processor (hardware) operations.Processor operations provides aconnection from a focal-point system to atarget system. Through NetView on thefocal-point system, processor operationsautomates operator and system consolesfor monitoring and recovering targetsystems. Also known as ProcOps.

Processor Resource/Systems Manager (PR/SM)The feature that allows the processor touse several operating system imagessimultaneously and provides logicalpartitioning capability. See also logicallypartitioned mode.

ProcOpsSee processor operations.

ProcOps Service Machine (PSM)The PSM is a CMS user on a VM hostsystem. It runs a CMS multitaskingapplication that serves as "virtualhardware" for ProcOps. ProOpscommunicates via the PSM with the VMguest systems that are defined as targetsystems within ProcOps.

product automationAutomation integrated into the base ofSA z/OS for the products CICS, DB2,IMS, TWS (formerly called features).

program operator interface (POI)A NetView facility for receiving VTAMmessages.

program to program interface (PPI)A NetView function that allows userprograms to send or receive data buffersfrom other user programs and to send

alerts to the NetView hardware monitorfrom system and application programs.

protocolIn SNA, the meanings of, and thesequencing rules for, requests andresponses used for managing the network,transferring data, and synchronizing thestates of network components.

proxy resourceA resource defined like an entry type APLrepresenting a processor operations targetsystem.

PSM See ProcOps Service Machine.

PU See physical unit.

R

RACF See Resource Access Control Facility.

remote systemA system that receives resource statusinformation from an SA z/OS focal-pointsystem. An SA z/OS remote system isdefined as part of the same SA z/OSenterprise as the SA z/OS focal-pointsystem to which it is related.

requesterA workstation from that user can log onto a domain from, that is, to the serversbelonging to the domain, and usenetwork resources. Users can access theshared resources and use the processingcapability of the servers, thus reducinghardware investment.

resourceAny facility of the computing system oroperating system required by a job ortask, and including main storage,input/output devices, the processing unit,data sets, and control or processingprograms.

In NetView, any hardware or softwarethat provides function to the network.

In SA z/OS, any z/OS application, z/OScomponent, job, device, or target systemcapable of being monitored or automatedthrough SA z/OS.

Resource Access Control Facility (RACF)A program that can provide data securityfor all your resources. RACF protects datafrom accidental or deliberateunauthorized disclosure, modification, ordestruction.

Glossary 335

resource groupA physically partitionable portion of aprocessor. Also known as a side.

Resource Measurement Facility (RMF)A feature of z/OS that measures selectedareas of system activity and presents thedata collected in the format of printedreports, System Management Facility(SMF) records, or display reports.

Resource Object Data Manager (RODM)In NetView for z/OS, a component thatprovides an in-memory cache formaintaining real-time data in an addressspace that is accessible by multipleapplications. RODM also allows anapplication to query an object and receivea rapid response and act on it.

resource tokenA unique internal identifier of an ESCONresource or resource number of the objectin the IODF.

restart automationAutomation provided by SA z/OS thatmonitors subsystems to ensure that theyare running. If a subsystem fails,SA z/OS attempts to restart it accordingto the policy in the automationconfiguration file.

Restructured Extended Executor (REXX)A general-purpose, high-level,programming language, particularlysuitable for EXEC procedures or programsfor personal computing, used to writecommand lists.

return codeA code returned from a program used toinfluence the issuing of subsequentinstructions.

REXX See Restructured Extended Executor.

REXX procedureA command list written with theRestructured Extended Executor (REXX),which is an interpretive language.

RMF See Resource Measurement Facility.

RODMSee Resource Object Data Manager.

S

SAF See Security Authorization Facility.

SA IOMSee System Automation for IntegratedOperations Management.

SAplexSAplex or "SA z/OS Subplex" is a termused in conjuction with a sysplex. In fact,a SAplex is a subset of a sysplex.However, it can also be a sysplex. For adetailed description, refer to "Using SAz/OS Subplexes" in IBM Tivoli SystemAutomation for z/OS Planning andInstallation.

SA z/OSSee System Automation for z/OS.

SA z/OS customization dialogsAn ISPF application through which theSA z/OS policy administrator definespolicy for individual z/OS systems andbuilds automation control data andRODM load function files.

SA z/OS customization focal point systemSee focal point system.

SA z/OS data modelThe set of objects, classes and entityrelationships necessary to support thefunction of SA z/OS and the NetViewautomation platform.

SA z/OS enterpriseThe group of systems and resourcesdefined in the customization dialogsunder one enterprise name. An SA z/OSenterprise consists of connected z/OSsystems running SA z/OS.

SA z/OS focal point systemSee focal point system.

SA z/OS policyThe description of the systems andresources that make up an SA z/OSenterprise, together with their monitoringand automation definitions.

SA z/OS policy administratorThe member of the operations staff whois responsible for defining SA z/OSpolicy.

SA z/OS satelliteIf you are running two NetViews on anz/OS system to split the automation andnetworking functions of NetView, it iscommon to route alerts to the NetworkingNetView. For SA z/OS to process alertsproperly on the Networking NetView, you


must install a subset of SA z/OS code,called an SA z/OS satellite on theNetworking NetView.

SA z/OS SDF focal point systemSee focal point system.

SCA In SA z/OS, system console A, the activesystem console for a target hardware.Contrast with SCB.

SCB In SA z/OS, system console B, thebackup system console for a targethardware. Contrast with SCA.

screen Deprecated term for panel.

screen handlerIn SA z/OS, software that interprets alldata to and from a full-screen image of atarget system. The interpretation dependson the format of the data on thefull-screen image. Every processor andoperating system has its own format forthe full-screen image. A screen handlercontrols one PS/2 connection to a targetsystem.

SDF See status display facility.

SDLC See synchronous data link control.

SDSF See System Display and Search Facility.

secondary systemA system is a secondary system for anapplication if it is defined to automationon that system, but the application is notnormally meant to be running there.Secondary systems are systems to whichan application can be moved in the eventthat one or more of its primary systemsare unavailable. SA z/OS does not startthe application on its secondary systems.

Security Authorization Facility (SAF)An MVS interface with which programscan communicate with an externalsecurity manager, such as RACF.

server A server is a workstation that sharesresources, which include directories,printers, serial devices, and computingpowers.

service language command (SLC)The line-oriented command language ofprocessor controllers or service processors.

service periodService periods allow the users toschedule the availability of applications. A

service period is a set of time intervals(service windows), during which anapplication should be active.

service processor (SVP)The name given to a processor controlleron smaller System/370 processors.

service thresholdAn SA z/OS policy setting thatdetermines when to notify the operator ofdeteriorating service for a resource. Seealso alert threshold and warningthreshold.

sessionIn SNA, a logical connection between twonetwork addressable units (NAUs) thatcan be activated, tailored to providevarious protocols, and deactivated, asrequested. Each session is uniquelyidentified in a transmission header by apair of network addresses identifying theorigin and destination NAUs of anytransmissions exchanged during thesession.

session monitorThe component of the NetView programthat collects and correlates session-relateddata and provides online access to thisinformation. The successor to NLDM.

shutdown automationSA z/OS-provided automation thatmanages the shutdown process forsubsystems by issuing shutdowncommands and responding to prompts foradditional information.

side A part of a partitionable CPC that can runas a physical partition and is typicallyreferred to as the A-side or the B-side.

Simple Network Management Protocol (SNMP)A set of protocols for monitoring systemsand devices in complex networks.Information about managed devices isdefined and stored in a ManagementInformation Base (MIB).

single imageA processor system capable of beingphysically partitioned that has not beenphysically partitioned. Single-imagesystems can be target hardwareprocessors.

Glossary 337

single-MVS environmentAn environment that supports one MVSimage. See also MVS image.

single-image (SI) modeA mode of operation for a multiprocessor(MP) system that allows it to function asone CPC. By definition, a uniprocessor(UP) operates in single-image mode.Contrast with physically partitioned (PP)configuration.

SLC See service language command.

SMP/ESee System ModificationProgram/Extended.

SNA See Systems Network Architecture.

SNA networkIn SNA, the part of a user-applicationnetwork that conforms to the formats andprotocols of systems network architecture.It enables reliable transfer of data amongend users and provides protocols forcontrolling the resources of variousnetwork configurations. The SNA networkconsists of network addressable units(NAUs), boundary function components,and the path control network.

SNMPSee Simple Network ManagementProtocol.

solicited messageAn SA z/OS message that directlyresponds to a command. Contrast withunsolicited message.

SSCP See system services control point.

SSI See subsystem interface.

start automationAutomation provided by SA z/OS thatmanages and completes the startupprocess for subsystems. During thisprocess, SA z/OS replies to prompts foradditional information, ensures that thestartup process completes within specifiedtime limits, notifies the operator ofproblems, if necessary, and bringssubsystems to an UP (or ready) state.

startupThe point in time that a subsystem orapplication is started.

status The measure of the condition oravailability of the resource.

status display facility (SDF)The system operations part of SA z/OSthat displays status of resources such asapplications, gateways, andwrite-to-operator messages (WTORs) ondynamic color-coded panels. SDF showsspool usage problems and resource datafrom multiple systems.

status focal-point systemSee focal point system.

steady state automationThe routine monitoring, both for presenceand performance, of subsystems,applications, volumes and systems.Steady state automation may respond tomessages, performance exceptions anddiscrepancies between its model of thesystem and reality.

structureA construct used by z/OS to map andmanage storage on a coupling facility.

subgroupA named set of systems. A subgroup ispart of an SA z/OS enterprise definitionand is used for monitoring purposes.

SubGroup entryA construct, created with thecustomization dialogs, used to representand contain policy for a subgroup.

subplexSee SAplex.

subsystemA secondary or subordinate system,usually capable of operating independentof, or asynchronously with, a controllingsystem.

In SA z/OS, an z/OS application orsubsystem defined to SA z/OS.

subsystem interface (SSI)The z/OS interface over which allmessages sent to the z/OS console arebroadcast.

support elementA hardware unit that providescommunications, monitoring, anddiagnostic functions to a central processorcomplex (CPC).

support processorAnother name given to a processor


controller on smaller System/370processors. See service processor.

SVP See service processor.

switch identifierThe switch device number (swchdevn),the logical switch number (LSN) and theswitch name

switchesESCON directors are electronic units withports that dynamically switch to routedata to I/O devices. The switches arecontrolled by I/O operations commandsthat you enter on a workstation.

symbolic destination name (SDN)Used locally at the workstation to relateto the VTAM application name.

synchronous data link control (SDLC)A discipline for managing synchronous,code-transparent, serial-by-bit informationtransfer over a link connection.Transmission exchanges may be duplex orhalf-duplex over switched or nonswitchedlinks. The configuration of the linkconnection may be point-to-point,multipoint, or loop. SDLC conforms tosubsets of the Advanced DataCommunication Control Procedures(ADCCP) of the American NationalStandards Institute and High-Level DataLink Control (HDLC) of the InternationalStandards Organization.

SYSINFO ReportAn RMF report that presents an overviewof the system, its workload, and the totalnumber of jobs using resources ordelayed for resources.

SysOpsSee system operations.

sysplexA set of z/OS systems communicatingand cooperating with each other throughcertain multisystem hardware components(coupling devices and timers) andsoftware services (couple data sets).

In a sysplex, z/OS provides the couplingservices that handle the messages, data,and status for the parts of a multisystemapplication that has its workload spreadacross two or more of the connectedprocessors, sysplex timers, coupling

facilities, and couple data sets (whichcontains policy and states forautomation).

A Parallel Sysplex is a sysplex thatincludes a coupling facility.

sysplex application groupA sysplex application group is a groupingof applications that can run on anysystem in a sysplex.

sysplex couple data setA couple data set that containssysplex-wide data about systems, groups,and members that use XCF services. Allz/OS systems in a sysplex must haveconnectivity to the sysplex couple dataset. See also couple data set.

Sysplex TimerAn IBM unit that synchronizes thetime-of-day (TOD) clocks in multipleprocessors or processor sides. ExternalTime Reference (ETR) is the z/OS genericname for the IBM Sysplex Timer (9037).

systemIn SA z/OS, system means a focal pointsystem (z/OS) or a target system (MVS,VM, VSE, LINUX, or CF).

System Automation for Integrated OperationsManagement

An outboard automation solution forsecure remote access tomainframe/distributed systems. TivoliSystem Automation for IntegratedOperations Management, previouslyTivoli AF/REMOTE, allows users tomanage mainframe and distributedsystems from any location.

The full name for SA IOM.

System Automation for OS/390The full name for SA OS/390, thepredecessor to System Automation forz/OS.

System Automation for z/OSThe full name for SA z/OS.

system consoleA console, usually having a keyboard anda display screen, that is used by anoperator to control and communicate witha system.

A logical device used for the operationand control of hardware functions (for

Glossary 339

example, IPL, alter/display, andreconfiguration). The system console canbe assigned to any of the physicaldisplays attached to a processor controlleror support processor.

In SA z/OS, the hardware system consolefor processor controllers or serviceprocessors of processors connected usingSA z/OS. In the SA z/OS operatorcommands and configuration dialogs, SCis used to designate the system consolefor a target hardware processor.

System Display and Search Facility (SDSF)An IBM licensed program that providesinformation about jobs, queues, andprinters running under JES2 on a series ofpanels. Under SA z/OS you can selectSDSF from a pull-down menu to see theresources’ status, view the z/OS systemlog, see WTOR messages, and see activejobs on the system.

System entryA construct, created with thecustomization dialogs, used to representand contain policy for a system.

System Modification Program/Extended(SMP/E)

An IBM licensed program that facilitatesthe process of installing and servicing anz/OS system.

system operationsThe part of SA z/OS that monitors andcontrols system operations applicationsand subsystems such as NetView, SDSF,JES, RMF, TSO, RODM, ACF/VTAM,CICS, IMS, and OPC. Also known asSysOps.

system services control point (SSCP)In SNA, the focal point within an SNAnetwork for managing the configuration,coordinating network operator andproblem determination requests, andproviding directory support and othersession services for end users of thenetwork. Multiple SSCPs, cooperating aspeers, can divide the network intodomains of control, with each SSCPhaving a hierarchical control relationshipto the physical units and logical unitswithin its domain.

System/390 microprocessor clusterA configuration that consists of central

processor complexes (CPCs) and mayhave one or more integrated couplingfacilities.

Systems Network Architecture (SNA)The description of the logical structure,formats, protocols, and operationalsequences for transmitting informationunits through, and controlling theconfiguration and operation of, networks.

T

TAF See terminal access facility.

target A processor or system monitored andcontrolled by a focal-point system.

target control taskIn SA z/OS, target control tasks processcommands and send data to targetsystems and workstations throughcommunications tasks. A target controltask (a NetView autotask) is assigned to atarget system when the target system isinitialized.

target hardwareIn SA z/OS, the physical hardware onwhich a target system runs. It can be asingle-image or physically partitionedprocessor. Contrast with target system.

target systemIn a distributed system environment, asystem that is monitored and controlledby the focal-point system. Multiple targetsystems can be controlled by a singlefocal-point system.

In SA z/OS, a computer system attachedto the focal-point system for monitoringand control. The definition of a targetsystem includes how remote sessions areestablished, what hardware is used, andwhat operating system is used.

task A basic unit of work to be accomplishedby a computer.

In the NetView environment, an operatorstation task (logged-on operator),automation operator (autotask),application task, or user task. A NetViewtask performs work in the NetViewenvironment. All SA z/OS tasks areNetView tasks. See also message monitortask, and target control task.


telecommunication lineAny physical medium, such as a wire ormicrowave beam, that is used to transmitdata.

terminal access facility (TAF)A NetView function that allows you tolog onto multiple applications either onyour system or other systems. You candefine TAF sessions in the SA z/OScustomization panels so you don't have toset them up each time you want to usethem.

In NetView, a facility that allows anetwork operator to control a number ofsubsystems. In a full-screen or operatorcontrol session, operators can control anycombination of subsystemssimultaneously.

terminal emulationThe capability of a microcomputer orpersonal computer to operate as if it werea particular type of terminal linked to aprocessing unit to access data.

thresholdA value that determines the point atwhich SA z/OS automation performs apredefined action. See alert threshold,warning threshold, and error threshold.

time of day (TOD)Typically refers to the time-of-day clock.

Time Sharing Option (TSO)An optional configuration of theoperating system that providesconversational time sharing from remotestations. It is an interactive service onz/OS, MVS/ESA, and MVS/XA.

Time-Sharing Option/Extended (TSO/E)An option of z/OS that providesconversational timesharing from remoteterminals. TSO/E allows a wide variety ofusers to perform many different kinds oftasks. It can handle short-runningapplications that use fewer sources aswell as long-running applications thatrequire large amounts of resources.

timers A NetView instruction that issues acommand or command processor (list ofcommands) at a specified time or timeinterval.

Tivoli Workload Scheduler (TWS)A family of IBM licensed products that

plan, execute and track jobs on severalplatforms and environments. Thesuccessor to OPC/A.

TOD Time of day.

token ringA network with a ring topology thatpasses tokens from one attaching deviceto another; for example, the IBMToken-Ring Network product.

TP See transaction program.

transaction programIn the VTAM program, a program thatperforms services related to theprocessing of a transaction. One or moretransaction programs may operate withina VTAM application program that isusing the VTAM application programinterface (API). In that situation, thetransaction program would requestservices from the applications programusing protocols defined by thatapplication program. The applicationprogram, in turn, could request servicesfrom the VTAM program by issuing theAPPCCMD macro instruction.

transitional automationThe actions involved in starting andstopping subsystems and applications thathave been defined to SA z/OS. This caninclude issuing commands andresponding to messages.

translating hostRole played by a host that turns aresource number into a token during aunification process.

triggerTriggers, in combination with events andservice periods, are used to control thestarting and stopping of applications in asingle system or a parallel sysplex.

TSO See Time Sharing Option.

TSO consoleFrom this 3270-type console you arelogged onto TSO or ISPF to use theruntime panels for I/O operations andSA z/OS customization panels.

TSO/E See Time-Sharing Option/Extended.

TWS See Tivoli Workload Scheduler.

U

Glossary 341

UCB See unit control block.

unit control block (UCB)A control block in common storage thatdescribes the characteristics of a particularI/O device on the operating system andthat is used for allocating devices andcontrolling I/O operations.

unsolicited messageAn SA z/OS message that is not a directresponse to a command.

uniform resource identifier (URI)A uniform resource identifier is a string ofcharacters used to identify a name of aweb resource. Such identification enablesinteraction with representations of theweb resource over the internet, usingspecific protocols.

user taskAn application of the NetView programdefined in a NetView TASK definitionstatement.

Using An RMF Monitor III definition. Jobsgetting service from hardware resources(processors or devices) are using theseresources. The use of a resource by anaddress space can vary from 0% to 100%where 0% indicates no use during aRange period, and 100% indicates that theaddress space was found using theresource in every sample during thatperiod.

V

view In the NetView Graphic Monitor Facility,a graphical picture of a network or part ofa network. A view consists of nodesconnected by links and may also includetext and background lines. A view can bedisplayed, edited, and monitored forstatus information about networkresources.

Virtual ServerA logical construct that appears tocomprise processor, memory, and I/Oresources conforming to a particulararchitecture. A virtual server can supportan operating system, associatedmiddleware, and applications. Ahypervisor creates and manages virtualservers.

Virtual Server CollectionA set of virtual servers that supports a

workload. This set is not necessarilystatic. The constituents of the collection atany given point are determined by virtualservers involved in supporting theworkload at that time.

virtual Server ImageA package containing metadata thatdescribes the system requirements, virtualstorage drives, and any goals andconstraints for the virtual machine {forexample, isolation and availability). TheOpen Virtual Machine Format (OVF) is aDistributed Management Task Force(DMTF) standard that describes apackaging format for virtual serverimages.

Virtual Server Image CaptureThe ability to store metadata and diskimages of an existing virtual server. Themetadata describes the virtual serverstorage, network needs, goals andconstraints. The captured information isstored as a virtual server image that canbe referenced and used to create anddeploy other similar images.

Virtual Server Image CloneThe ability to create an identical copy(clone) of a virtual server image that canbe used to create a new similar virtualserver.

Virtual Storage Extended (VSE)A system that consists of a basicoperating system (VSE/AdvancedFunctions), and any IBM supplied anduser-written programs required to meetthe data processing needs of a user. VSEand the hardware that it controls form acomplete computing system. Its currentversion is called VSE/ESA.

Virtual Telecommunications Access Method(VTAM)

An IBM licensed program that controlscommunication and the flow of data in anSNA network. It provides single-domain,multiple-domain, and interconnectednetwork capability. Its full name isAdvanced Communications Function forthe Virtual Telecommunications AccessMethod. Synonymous with ACF/VTAM.

VM Second Level Systems SupportWith this function, Processor Operationsis able to control VM second level


|||||||

systems (VM guest systems) in the sameway that it controls systems running onreal hardware.

VM/ESAVirtual Machine/Enterprise SystemsArchitecture. Its current version is calledz/VM.

volumeA direct access storage device (DASD)volume or a tape volume that serves asystem in an SA z/OS enterprise.

VSE See Virtual Storage Extended.

VTAMSee Virtual Telecommunications AccessMethod.

W

warning thresholdAn application or volume service valuethat determines the level at whichSA z/OS changes the associated icon inthe graphical interface to the warningcolor. See alert threshold.

workstationIn SA z/OS workstation means thegraphic workstation that an operator usesfor day-to-day operations.

write-to-operator (WTO)A request to send a message to anoperator at the z/OS operator console.This request is made by an applicationand is handled by the WTO processor,which is part of the z/OS supervisorprogram.

write-to-operator-with-reply (WTOR)A request to send a message to anoperator at the z/OS operator consolethat requires a response from theoperator. This request is made by anapplication and is handled by the WTOprocessor, which is part of the z/OSsupervisor program.

WTO See write-to-operator.

WTORSee write-to-operator-with-reply.

WWV The US National Institute of Standardsand Technology (NIST) radio station thatprovides standard time information. Asecond station, known as WWVB,provides standard time information at adifferent frequency.

X

XCF See cross-system coupling facility.

XCF couple data setThe name for the sysplex couple data setprior to MVS/ESA System ProductVersion 5 Release 1. See also sysplexcouple data set.

XCF groupA set of related members that amultisystem application defines to XCF. Amember is a specific function, or instance,of the application. A member resides onone system and can communicate withother members of the same group acrossthe sysplex.

XRF See extended recovery facility.

Z

z Appliance Container Infrastructure (zACI) IBM z Systems partitions, activated to runin zACI operating mode, provide thebasic infrastructure run-time anddeployment support for firmware orsoftware based appliances, such aszAware or z/VSE VNA.

z/OS An IBM mainframe operating system thatuses 64-bit real storage. See also BaseControl Program.

z/OS componentA part of z/OS that performs a specificz/OS function. In SA z/OS, componentrefers to entities that are managed bySA z/OS automation.

z/OS subsystemSoftware products that augment the z/OSoperating system. JES and TSO/E areexamples of z/OS subsystems. SA z/OSincludes automation for some z/OSsubsystems.

z/OS systemA z/OS image together with its associatedhardware, which collectively are oftenreferred to simply as a system, or z/OSsystem.

z196 See IBM Enterprise 196 (z196).

zAAP See IBM System z Application AssistProcessor (zAAP).

zBX See IBM zEnterprise BladeCenterExtension (zBX).

Glossary 343

zBX bladeSee IBM zEnterprise BladeCenterExtension (zBX) blade.

zCPC The physical collection of main storage,central processors, timers, and channelswithin a zEnterprise mainframe. Althoughthis collection of hardware resources ispart of the larger zEnterprise centralprocessor complex, you can apply energymanagement policies to zCPC that aredifferent from those that you apply to anyattached IBM zEnterprise BladeCenterExtension (zBX) or blades. See also centralprocessor complex.

zIIP See IBM System z Integrated InformationProcessor (zIIP).

zEnterpriseSee IBM zEnterprise System (zEnterprise).


Index

Special characters"hung" command recovery 139

Aaccessibility xiiiactive connector 135active health monitoring 45adding

application to automation 1processor operations message to

automation 95additional automation operator IDs 153additional SA z/OS automation

procedures, programming 7advanced automation options

exits 170external global variables 241, 242

alertscommunication flow 75enabling 76enabling at alert ID level 78enabling at system level 77enabling globally 77enabling with Inform List 77enabling with INGCNTL 77, 78notification 75overview 75

alternate CDS 133turning into primary CDS 134

alternate CDS recoverycustomizing 134

alternate couple data setspecifying 147

AMRF buffer shortage processing 192AOCMSG call 16AOCMSG generic routine 10AOCQRY common routine

automation availability 9message automation 22

AOCTRACEuse in testing 19use in traces 20

AOCUPDT commandand AOFEXSTA exit 173

AOCUPDT common routineto update status information 10

AOF_AAO_AAREQUEST_MAX_WAIT 242AOF_AAO_AOCUPDT_LOGGING 242AOF_AAO_GWPING_RETRY 242AOF_AAO_HW_VALIDATION 242AOF_AAO_INJECT_NOFORCE_REQ 242AOF_AAO_IPL_COMPLETE_MSG 242AOF_AAO_ISQ_KALIST 242AOF_AAO_MSG_EHKVAR 242AOF_AAO_MVSTAPEMON 242AOF_AAO_OMVS_SHUTDOWN 242AOF_AAO_RDS_TSO_DSN 242AOF_AAO_RDS_TSO_RACFHLQ 242AOF_AAO_RETENTIONPERIOD 242

AOF_AAO_SDF_MSGDEL_OPERID 242AOF_AAO_SDFCSAPLEX 242AOF_AAO_SDFCxxx.* 242AOF_AAO_SDFROOT_LISTn 242AOF_AAO_SDFROOT.* 242AOF_AAO_SEC_PPIAUTH 242AOF_AAO_SHUTDOWN_STOPAPPL 242AOF_AAO_SHUTSYS_OLD 242AOF_AAO_TRANRERUN 242AOF_AAO_TWS_CMD_OUTPUT_NETLOG 242AOF_AAO_TWS_ERRMSG 242AOF_AAO_TWS_MAX_WAIT_TIME 242AOF_AAO_TWS_RESYSPLEX 242AOF_ASSIGN_JOBNAME 242AOF_E2E_EAS_PPI 242AOF_E2E_EVT_RETRY 242AOF_E2E_EXREQ_NETLOG 242AOF_E2E_TKOVR_TIMEOUT 242AOF_EMCS_AUTOTASK_ASSIGNMENT 242AOF_EMCS_CN_ASSIGNMENT 242AOF_INIT_MCSFLAG 242AOF_INIT_ROUTCDE 242AOF_INIT_SYSCONID 242AOF_PRODLVL 241AOF.0DEBUG 241AOF.0TRACE 241AOFACFINIT 242AOFAOCCLONE 241AOFARMQUERYRETRYS 242AOFARMQUERYWAIT 242AOFBFP 241AOFCFP 241AOFCNMASK 242AOFCNMASK_EXTENDED 242AOFCOMPL 241AOFCONFIRM global variable 187AOFDEBUG 241AOFDEBUG global variable 20AOFDEFAULT_TARGET 242AOFDOM 283, 284AOFEXC00 exit 183AOFEXC01 exit 183AOFEXC02 exit 183AOFEXC03 exit 183AOFEXC04 exit 183AOFEXC05 exit 183AOFEXC06 exit 184AOFEXC07 exit 184AOFEXC08 exit 184AOFEXC09 exit 184AOFEXC11 exit 184AOFEXC13 exit 185AOFEXC14 exit 185AOFEXC15 exit 185AOFEXC16 exit 185AOFEXC17 exit 185AOFEXC18 exit 185AOFEXC19 exit 186AOFEXC20 exit 186AOFEXC21 exit 186AOFEXC22 exit 187

AOFEXC23 exit 187AOFEXC24 187AOFEXC25 187AOFEXDEF exit 170AOFEXI01 exit 171AOFEXI02 exit 171AOFEXI03 exit 171AOFEXI04 exit 171AOFEXI05 exit 171AOFEXI06 exit 172AOFEXINT exit 172, 188, 242AOFEXPLAIN_USER 242AOFEXSTA exit 173AOFEXX02 exit 174AOFEXX03 exit 174AOFEXX04 exit 175AOFEXX05 175AOFEXX15 exit 175AOFINITIALSTARTTYP 241AOFINITREPLY 242AOFJESPREFX 241AOFJRYCMD description 123AOFLOCALHOLD 242AOFMATLISTING 242AOFMSGST 34AOFMSGSY 283AOFOPCCMDMSG 242AOFPAUSE 242AOFRESTARTALWAYS 242AOFRJ3MN monitoring routine 62AOFRJ3RC monitoring routine 64AOFRMTCMDWAIT 242AOFRPCWAIT 242AOFRSA01 automation routine 198AOFRSA02 automation routine 199AOFRSA03 automation routine 201AOFRSA08 automation routine 203AOFRSA0C automation routine 205AOFRSA0E automation routine 208AOFRSA0G automation routine 209AOFRSD01 automation routine 220AOFRSD07 automation routine 211AOFRSD09 automation routine 212AOFRSD0F automation routine 213AOFRSD0G automation routine 215AOFRSD0H automation routine 216AOFSENDALERT 242AOFSERXINT 242AOFSETSTATEOVERRIDE 255AOFSETSTATESCOPE 255AOFSETSTATESTART 255AOFSHUTDELAY 242AOFSMARTMAT 242AOFSPOOLFULLCMD 242AOFSPOOLSHORTCMD 242AOFSTATUSCMDSEL 242AOFSUBSYS 241AOFSYS 283, 284AOFSYSNAME 241AOFSYSPLEX 241AOFSYSPLEXGROUP 241


AOFSYSTEM 241AOFUPDAM 242AOFUPDRODM 242AOFUSSWAIT 242application

adding to automation 1health status 41

application monitor status 41application monitoring 41application type IMAGE, defining 149application, VTAM, defining to

SA z/OS 159applications, z/OS UNIX 107ARM 282ASCB chaining

and global variables 248ASFUSER command 22assist mode

for testing automation procedures 20overview 19

assumptions, health monitoring withOMEGAMON 51

asynchronous hardware commands,using pipes and ISQCCMD for 101

AT actions, defining for messageautomation 26

AT buildconcept for message automation 31message automation 31

AT load, message automation 32Autodiscovery Mapper Files 293Autodiscovery Report Format 299Autodiscovery, building configuration

control data 238Autodiscovery, Components

Overview 228Autodiscovery, Extended Automated

Modelling 236Autodiscovery, Finalizing the Target

Policy 237Autodiscovery, Process Overview 230automated resources, z/OS UNIX

Automation 110Automated System Resource

Discovery 227Automatic Restart Manager 281

defining element name 281MOVE group for 282

automatingauxiliary storage shortage

recovery 152enqueues, long running 151IXC102A message 141IXC402D message 141Linux console messages 94Linux console messages, case

sensitive 94Linux console messages, restrictions

and limitations 95Linux console messages, security

considerations 95long running enqueues 151message IXC102A 149message IXC402D 149USS resources 107

automating processor operationscontrolled resources 91

automationadding an application to 1advanced functions 242extending 7messages 25sysplex, enabling 133

automation control filedefining SDF 280reload action exit 188reload permission exit 187

automation flag exitssample 178

automation networks 155automation operator IDs

additional 153automation procedure

developing messages 15automation procedures

calling 7creating 7debugging 19description 7example 16external code 11global variable names 23initializing 9installing 19making generic 14programming recommendations 22REXX coding example 21structure of 8testing 19use of commands in 7use of routines in 7using AOCTRACE 20writing your own 7

automation processingperforming 10

automation routineAMRF buffer shortage

processing 192AOFRSA01 198AOFRSA02 199AOFRSA03 201AOFRSA08 203AOFRSA0C 205AOFRSA0E 208AOFRSA0G 209AOFRSD07 211AOFRSD09 212AOFRSD0F 213AOFRSD0G 215AOFRSD0H 216drain processing prior to JES2

shutdown 192HASP099 218IMS transaction recovery 192INGRCJSP 220INGRMJSP 219INGRTAPE 222INGRX740 223INGRYSHU 225introduction 189LOGREC data set processing 190processing 190SMF data set processing 190SVC dump processing 191

automation setup, definitions for 108automation status file

coding your own information 22using commands 11

automation table 35auxiliary storage shortage recovery 142

automating 152defining local page data set 152defining the handling of jobs 152

availability, reporting 83INGPUSMF utility 86INGPUSMF utility JCL 86INGPUSMF utility JCL, user

options 87INGPUSMF utility outrput 86INGPUSMF utility return codes 87overview 83resource lifecycle 83SMF record layout 84writing to DB2 88

BBASEOPER 242batch command interface

JCL 123batch job

command continuation 122command information retrieval 123command output redirection 122command statement syntax 122sample JCL 122submitting NetView commands

from 121valid command types 122

buildingnew automation definitions 99

Ccalling

automation procedures 7captured messages

defining for message automation 27cascades 286case sensitive, Linux console

messages 94CDEMATCH common routine 22CDS 133CF 135CFRM couple data set 135, 147CFRM policy 135CHKTHRES automation procedure 11CICS health monitoring 58, 60CICS link monitoring 60CICS monitoring

component overview 60defining monitor resources 61VOST management 61

CICSPlex monitoring 60clone ID

Automatic Restart Manager 281clone ID, Automatic Restart

Manager 242CMD actions, defining for message

automation 26


CNMCMDU member 19coding information in automation status

file 22command flooding recovery 140command output redirection batch

job 122Command Receiver 119command, SDFCONF 277commands

NetView 121processor operations 15use in automation procedures 7

commands, defining for long runningenqueues 152

commands, monitor resources 43common automation items, defining 153common global variables 11, 241communication flow

alerts 75connecting

system to processor 146connector

active 135failed persistent 135

continuous availability, couple data setenabling 147ensuring 134

couple data set 133alternate CDS 133alternate CDS, recovery of 134alternate, specifying 147CFRM 147enabling continuous availability

of 147ensuring continuous availability

of 134managing 133policy 133primary CDS 133SYSPLEX 147

coupling facility 135coupling facility, managing 135creating automation procedures 7customization dialog exits 178

invocation 182customization of z/OS UNIX

resources 108customize automation

for processor operations 12for system operations 10

customizingalternate CDS recovery 134hung command recovery 141IXC102A message automation 142IXC402D message automation 142LINUX target systems 103MVS target systems 104proxy resource automation 92SDF 265system logger recovery 135system to use Parallel Sysplex

enhancements 153target systems 103VM target systems 104VSE target systems 105

DDB2, writing SMF report to 88debugging

automation procedures 19NetView facilities 21z/OS UNIX Automation 118

definingactions for message automation 25application type IMAGE 149AT actions for message

automation 26AT entry placement 26captured messages for message

automation 27CMD actions for message

automation 26commands for long running

enqueues 152common automation items 153conditions for AT entries 26gateway sessions 156handling of jobs for auxiliary storage

shortage recovery 152IEADMCxx symbols for long running

enqueues 152IMAGE application type 149local page data set for auxiliary

storage shortage recovery 152logical partitions 146logical sysplex 147message revision table entries 30override for message automation 28physical sysplex 147processor 146, 149REPLY actions for message

automation 26resources for long running

enqueues 151SDF focal point system 155SDF in automation control file 280snapshot intervals for long running

enqueues 152started task job name 153status messages for message

automation 26SYSPLEX policy item 147system 146TAF fullscreen sessions 156temporary data set HLQ 153VTAM application to SA z/OS 159

definitions for automation setup 108definitions for z/OS UNIX

resources 108deletion of processed WTO(R)s from

SDF 191developing messages for automation

procedures 15directory extent 135disability xiiiDISPEVT_WAIT 255DISPEVTS_WAIT 255DISPGW_COLUMNS 255DISPMTR_COLUMNS 255DISPSTAT_COLUMNS 255DISPTRG_WAIT 255drain processing prior to JES2

shutdown 192

DSIPARM data set 19

Eelement name, Automatic Restart

Managerdefining 281

element namesin Automatic Restart Manager 281

element names in Automatic RestartManager 242

enablingalerting 76alerting with INGCNTL 78alerting, with Inform List 77alerting, with INGCNTL 77continuous availability of Couple Data

Sets 147sysplex automation 133system removal 148WTOR(R) buffer shortage

recovery 148enabling Relational Data Services 127enabling, command receiver 119ENQs 138enqueues 138

long running, automating 151long running, customizing recovery

of 141long running, handling 138

environmental setup exits 172error codes 11events, resource lifecycle 83EVIRYDP0_COLUMNS 255example automation procedure 16examples of INGUSS command 112exits 188

AOFEXC00 183AOFEXC01 183AOFEXC02 183AOFEXC03 183AOFEXC04 183AOFEXC05 183AOFEXC06 184AOFEXC07 184AOFEXC08 184AOFEXC09 184AOFEXC11 184AOFEXC13 185AOFEXC14 185AOFEXC15 185AOFEXC16 185AOFEXC17 185AOFEXC18 185AOFEXC19 186AOFEXC20 186AOFEXC21 186AOFEXC22 187AOFEXC23 187AOFEXC24 187AOFEXC25 187AOFEXDEF 170AOFEXI01 171AOFEXI02 171AOFEXI03 171AOFEXI04 171AOFEXI05 171

Index 347

exits (continued)AOFEXI06 172AOFEXINT 172, 188AOFEXSTA 173AOFEXX02 174AOFEXX03 174AOFEXX04 175AOFEXX05 175AOFEXX15 175BUILD processing 178CONVERT processing 181COPY processing 180customization dialog exits 178DELETE processing 180environmental setup exits 172flag exits 175IMPORT functions 181INGEAXIT 175INGEX01 178INGEX02 178INGEX03 180INGEX04 180INGEX05 180INGEX06 180INGEX07 181INGEX08 181INGEX09 181INGEX12 181INGEX14 181INGEX16 181INGEX17 181INGEX18 181pseudo-exits 187RENAME functions 181sample automation flag exits 178static exits 173status change commands 173subsystem up at initialization

commands 188testing 188

EXPLAIN 250extended status command support

introduced 28policy definitions 29

extending automation 7external code, automation procedures 11external common global variables 241EXTSTART status 242, 281

Ffailed persistent connector 135failed system, isolation of 141file manager commands 11file monitoring, z/OS UNIX

Automation 112flag exits 175focal point system definition 155

Ggateway

inbound 156outbound 156

gateway sessionsdefining 156

GDPS environment, shutting down z/OSsystems in 161

genericautomation 35, 242

generic automation procedures 14global variable names, for automation

procedures 23guest machines, processor operations

support 102guest target systems

LINUX 102LINUX, user logon 102MVS 102MVS, NIP console 102MVS, NIP messages 102MVS, problem determination

mode 102ProcOps Service Machine 102VSE 102

Hhardware commands

asynchronous, using pipes andISQCCMD for 101

synchronous, using pipes andISQCCMD for 100

HASP099 automation routine 218health monitoring

active 45event-based 46overview 42passive 46

health monitoring, OMEGAMONexceptions

introduction 55health monitoring, OMEGAMON XE

situationsintroduction 58

health status return codes 45health-based automation using

OMEGAMONprogramming techniques 48recommendations 57recovery techniques 44

how to automate USS resources 107hung command recovery,

customizing 141

IIDENT 22IEADMCxx symbols, defining

for long running enqueues 152IMAGE application type, defining 149IMS automation, monitoring 66IMS transaction recovery 192inbound gateway 156INCLUDE statement 279INGAMS_COLUMNS 255INGAUTO_INTERVAL 255INGCF command 135INGCICS_CORRWAIT 242INGDLG 182INGEAXIT exit 175INGEI004 member 101

INGEVENT_WAIT 255INGEX01 178INGEX02 178INGEX03 180INGEX04 180INGEX05 180INGEX06 180INGEX07 181INGEX08 181INGEXEC_RESP 255INGEXEC_SELECT 255INGEXEC_TIMEOUT 255INGEXEC_WAIT 255INGGROUP_WAIT 255INGHIST_MAX 255INGHIST_WIMAX 255INGIMS_CMDWAIT 255INGIMS_CORRWAIT 242INGIMS_REQ 255INGINFO_WAIT 255INGKLUP_WAIT 255INGLIST_COLUMNS 255INGLIST_WAIT 255INGLKUP_TIMEOUT 255INGMON_WAIT 255INGMON, programming techniques 48INGMOVE_WAIT 255INGMSG01 34INGMSGSA 34INGMTRAP monitor command 55INGOMX API 52INGOPC_MULTIPLIER 242INGPAC_SHOWNOLIMT 242INGPAC_WAIT 255INGPUSMF utility

introduced 86JCL 86JCL, user options 87output 86return codes 87

INGRCJSP automation routine 220INGRELS_SHOW 255INGRELS_WAIT 255INGREQ_EXPIRE 255INGREQ_INTERRUPT 255INGREQ_ORIGINATOR 242INGREQ_OVERRIDE 255INGREQ_PRECHECK 255INGREQ_PRI 255INGREQ_PRI.E2EMGR 255INGREQ_REMOVE 255INGREQ_REMOVE.START 255INGREQ_REMOVE.STOP 255INGREQ_RESTART 255INGREQ_SCOPE 255INGREQ_SOURCE 255INGREQ_TIMEOUT 255INGREQ_TYPE 255INGREQ_VERIFY 255INGREQ_WAIT 255INGRMJSP automation routine 219INGRPT_WAIT 255INGRTAPE automation routine 222INGRUN_CMT 255INGRUN_MULT 255INGRUN_OVERRIDE 255INGRUN_PERSISTENT 255


INGRUN_PRI 255INGRUN_REQ 255INGRUN_RUNMODE 255INGRUN_RUNRES 255INGRUN_SYSTEM 255INGRUN_TARGET 255INGRUN_TYPE 255INGRUN_VERIFY 255INGRUN_WAIT 255INGRX740 automation routine 223INGRYSHU automation routine 225INGSCHED_WAIT 255INGSET_VERIFY 255INGSET_WAIT 255INGSTX_WAIT 255INGTRIG_WAIT 255INGUSS command 112

examples 112INGVOTE_EXCLUDE 255INGVOTE_SOURCE 255INGVOTE_STATUS 255INGVOTE_VERIFY 255initialization processing,

AOFSERXINT 242initializing automation procedures 9installing

automation procedure 19integration of z/OS UNIX System

Services 107ISQCCMD

using for asynchronous hardwarecommands 101

using for synchronous hardwarecommands 100

ISQEXEC command 13, 95ISQOVRD 96ISQOVRD command 14ISQXLOC command 14ISQXMON command 95ISQXUNL command 14IXC102A message

automating 149automation of 141customizing automation of 142

IXC402D messageautomating 149automation of 141customizing automation of 142

IXCARM macro invocations 281IXCMIAPU 281

JJES2 spool files

Joblog Monitoring 67JES2 spool monitoring 65JES3 monitoring 61job handling, defining for auxiliary

storage shortage recovery 152job/ASID definitions, making

for long running enqueues 151Joblog Monitoring 67

setting up 67

Kkeyboard xiii

Llayout, SMF record 84Linux console connection to NetView 94Linux console messages

automating 94case sensitive 94restrictions and limitations 95security considerations 95

LINUX guest target systems, userlogon 102

LINUX target systems, customizing 103local page data set, defining

for auxiliary storage shortagerecovery 152

log stream 134log stream data set 134logical partition

defining 146logical sysplex, defining 147LOGR couple data set 134, 135LOGREC data set processing 190long running enqueues

automating 151defining commands 152defining IEADMCxx symbols 152defining resources 151defining snapshot intervals 152handling 138making job/ASID definitions 151

Mmaking generic automation

procedures 14making job/ASID definitions

for long running enqueues 151managing

couple data set 133coupling facilities 135system logger 134

master automation tables 34multiple 34

member, INGEI004 101message

forwarding 95forwarding path, defining 155ISQ900I 95ISQ901I 95IXC102A, automation of 141IXC402D, automation of 141testing 95, 98

message automation 25AT build 31AT load 32AT/MRT build concept 31defining actions 25defining AT actions 26defining captured messages 27defining CMD actions 26defining conditions for AT entries 26defining message revision table

entries 30

message automation (continued)defining overrides 28defining REPLY actions 26defining status messages 26Linux console messages 94Linux console messages, case

sensitive 94Linux console messages, restrictions

and limitations 95Linux console messages, security

considerations 95overview 25preparing for processor operations

resources 93specifying entry placement 26use of symbols 25

message automation for processoroperations resources 91

message presentation 284message revision table, defining

entries 30message testing 98messages

automation 25classifications 33developing for automation

procedures 15trapping UNIX syslogd 118

minor resourcesresource name 177

monitor command, INGMTRAP 55monitor resources 41

commands 43defining for CICS monitoring 61defining for OMEGAMON XE

situations 58monitor routine

writing your own 41monitoring

applications 41CICS health 60CICS link 60CICSPlex 60health with OMEGAMON 50health, active 45health, event-based 46health, overview 42health, passive 46IMS automation 66JES3 components 61observed status 41using OMEGAMON XE situations 58

monitoring routinesAOFRJ3MN 62AOFRJ3RC 64

monitoring routines for z/OS UNIXresources 108

MOVE group for Automatic RestartManager 282

MOVED statusAutomatic Restart Manager 281automation 281

MPF list 99MRT build

concept for message automation 31MTR 41

Index 349

MVS Automatic Restart Managerclone ID 242element names 242global variables 242

MVS guest target systemsNIP console 102NIP messages 102problem determination mode 102

MVS target systems, customizing 104MVSESA.RELOAD.ACTION minor

resource 188MVSESA.RELOAD.CONFIRM flag 187MVSESA.RELOAD.CONFIRM minor

resource 187

NNetView

generic automation table entries 35Linux console connection to 94testing and debugging facilities 21

NetView automation tableadding processor operations messages

to 95AOFMSGSY 283defining conditions for AT entries 26fragments 283generic entries 35integrating 34ISQEXEC 13, 95ISQOVRD 14ISQXLOC 14ISQXMON 95ISQXUNL 14master automation tables 34merging entries 98multiple master automation tables 34production 98reloading tables 99sample entry 96samples 33specifying entry placement 26structure 33user-written statements 34

NetView commandsexecuting on a different NetView 123submitting from a batch job 121

networks automationdefinition process 155

new automation definitionsbuilding 99

notificationalerts 75

notifications 10

Oobserved status

monitoring 41OMEGAMON

assumptions 51exception analysis 50exceptions, health monitoring 55health monitoring 58health monitoring with 50

OMEGAMON (continued)health-based automation,

programming techniques 48health-based automation,

recommendations 57health-based automation, recovery

techniques 44interaction 52monitoring, overview 50session management, INGMTRAP 55session management, INGOMX 52usage scenario 56

OMEGAMON XE situation monitoringdefining monitor resources 58overview 58

OMEGAMON XE situations,monitoring 58

operator cascades 286outbound gateway 156override

defining for message automation 28overview

alerts 75message automation 25monitoring with OMEGAMON 50

Ppanels

DISPACF 210, 214, 216INGTHRES 207JES2 214, 217LOGREC 200SMF 202SYSLOG 203, 204

passive, event-based healthmonitoring 46

persistent connection 135persistent structure 135physical sysplex, defining 147PIPE labels 123pipes

using for asynchronous hardwarecommands 101

using for synchronous hardwarecommands 100

policyCFRM 135couple data set 133

preference list 135Preloader Reports 299primary CDS 133problem determination mode

MVS guest target systems 102process monitoring, z/OS UNIX

Automation 110processing, WTOR 165processor

defining 146, 149PROCESSOR INFO policy item

using 146processor operations

guest machines support 102processor operations command

messages 97processor operations commands 15

processor operations controlled resources,automating 91

processor operations resource 91processor operations resource message

automation 91ProcOps Service Machine 102

guest target systems 102programming

additional SA z/OS automationprocedures 7

recommendations for automationprocedures 22

programming recommendationsautomation procedures 22

proxy resource 92proxy resources

customizing automation for 92shutdown considerations 93startup considerations 93

pseudo-exits 187PSM 102

RRDS Table Editor 130rebuild 135

system-managed 135user-managed 135

recommendationsprogramming, for automation

procedures 22recovery

"hung" command 139alternate CDS 134alternate CDS, customizing 134auxiliary storage shortage 142auxiliary storage shortage,

automating 152command flooding 140handling long-running enqueues 138long running enqueues,

customizing 141system log failure 191system logger, customizing 135system logger, directory shortage 135WTO(R) buffer shortage 137WTOR(R) buffer shortage,

enabling 148recovery time, reporting 83

INGPUSMF utility 86INGPUSMF utility JCL 86INGPUSMF utility JCL, user

options 87INGPUSMF utility outrput 86INGPUSMF utility return codes 87overview 83resource lifecycle 83SMF record layout 84writing to DB2 88

redirection, batch job commandoutput 122

Relational Data Services 127reload action exit 188reload permission exit 187RELOAD.ACTION flag 188RELOAD.CONFIRM flag 187reloading NetView automation table 99


REPLY actionsdefining for message automation 26

reporting, availability and recoverytime 83

resolvingWTO(R) buffer shortages 137

resource lifecycle, events 83resources, defining for long running

enqueues 151restrictions and limitations, Linux console

messages 95return codes, health status 45REXX coding example 21REXX PARSE 22REXX trace type 20routines

use in automation procedures 7

SSA IOM 75SA z/OS

commands ISQXIPM andISQCMMT 12

SA z/OS, defining VTAM applicationto 159

sampleautomation tables 33

scenarioOMEGAMON 56

SDFand specific problems 271components 273customizing 265customizing initialization

parameters 280defining hierarchy 275defining in automation control

file 280defining in customization dialog 280defining panels 276definition process 275for multiple systems 273how it works 265panels

definition 272, 276types 265

starting and stopping 274status descriptors 266tree structures 267

SDFCONF command 277second level systems, VM support 102security considerations, Linux console

messages 95serialize command processing 13session management

OMEGAMON, INGMTRAP 55OMEGAMON, INGOMX 52

setting up z/OS UNIX automation 108example 113

SFM 141shortcut keys xiiishutdown considerations, proxy resource

automation 93shutting down z/OS Systems from GDPS

environment 161SMF data set processing 190

SMF report, writing to DB2 88snapshot intervals, defining for long

running enqueues 152spool monitoring, JES2 65start definitions for z/OS UNIX

resources 112started task job name

defining 153startup considerations, proxy resource

automation 93status change commands 173status command support, extended

introduced 28policy definitions 29

status descriptors 268chaining to status components 269propagating 270

status information 10status messages

defining for message automation 26stop definitions for z/OS UNIX

resources 112structure 135

allocation 135automation procedures, of 8deallocation 135duplexing 135persistent 135preference list 135rebuild 135system-managed rebuild 135user-managed rebuild 135

SUBSAPPL 22SUBSJOB 22SUBSTYPE 22subsystem

adding to automation 1up at initialization commands 188

SVC dump processing 191symbols

use with message automation 25synchronous hardware commands, using

pipes and ISQCCMD for 100syntax, batch job command

statement 122SYSLOG processing 190syslogd messages, trapping 118sysplex automation

enabling 133SYSPLEX couple data set 147Sysplex Failure Management 141sysplex functions 133

switching on and off 153SYSPLEX policy item

defining 147system

connecting to processor 146defining 146

system log failure recovery 191system logger

directory extent 135log stream 134log stream data set 134LOGR couple data set 135managing 134recovery, customizing 135recovery, directory shortage 135

system operations control files 99system removal 141

enabling 148system-managed rebuild 135

TTAF fullscreen sessions

defining 156target systems, customizing 103task global variables 11TCP port monitoring, z/OS UNIX

Automation 112temporary data set HLQ

defining 153Terminal Access Facility 156testing

automation procedures 19messages 98more information 22NetView facilities 21

testing exits 188Topology Manager 289transaction recovery

IMS 192TRAP AND WAIT processing 101trapping UNIX syslogd messages 118

UUNIX Automation

automated resources 110debugging 118file monitoring 112hints and tips 117process monitoring 110setting up 108setup example 113TCP port monitoring 112

UNIX resourcescustomization of 108definitions for 108monitoring routines for 108start and stop definitions 112

UNIX syslogd messages, trapping 118UNIX System Services, integration 107user exits 169

static exits 173user logon, LINUX guest target

systems 102user-managed rebuild 135using

PROCESSOR INFO policy item 146USS resources

automating 107

VVM second level systems support 102VM target systems, customizing 104VOST management, CICS monitoring 61VSE guest target systems 102VSE target systems, customizing 105VTAM application, defining to

SA z/OS 159

Index 351

WWTO(R)

automation routinedeletion of processed WTO(R)s

from SDF 191processed, deletion from SDF 191

WTO(R) buffer 137WTOR processing 165WTOR(R) buffer shortage

recovery, enabling 148

Zz/OS systems, shutting down in a GDPS

Environment 161z/OS UNIX applications 107

infrastructure overview 107z/OS UNIX Automation

automated resources 110debugging 118file monitoring 112hints and tips 117process monitoring 110setting up 108setup example 113TCP port monitoring 112

z/OS UNIX resourcescustomization of 108definitions for 108monitoring routines for 108start and stop definitions 112

z/OS UNIXSystem Services, integrationof 107


IBM®

Printed in USA

SC34-2715-00

system automation for z/os: customizing and …...syslog pr ocessing ..... . 190 system log failur e...

Documents