pbs analytics 12.2 administrator's guide

PBS Works is a division of

PBS AnalyticsTM 13.0

Administrator’s Guide

PBS Analytics 13.0 Administrator’s Guide

Updated: July 29, 2015.

Intellectual Property Rights Notice:

Copyrights, Trademarks, Trade Secrets, Patents & Third Party Software Licenses

Altair® PBS Works™

Enabling On-Demand Computing™

Copyright© 1994-2015 Altair Engineering Inc. All Rights Reserved.

Special Notice: Pre-release versions of Altair software are provided ‘as is’, without warranty of any kind. Usage is strictly limited to non-production purposes.

Compute Manager™ 2007-2015; Display Manager™ 2007-2015; PBS™ 1994-2015; PBS Professional® 1994-2015; PBS Application Services™ 2008-2015; PBS Analytics™ 2007-2015; and PBS Desktop™ 2007-2012; PBS Portal™ 2007-2011; e-BioChem ™ 2007-2013; e-Compute™ 2000-2007; e-Render™ 2006-2010; OpenPBS® 1994-2003 and Personal PBS® 2007-2012.

Other Altair software applications include:

Altair Simulation Cloud Suite: Simulation Manager™ 2003-2015; Compute Manager™ 2003-2015; Display Manager™ 2003–2015; and Process Manager™ 2003-2015.

Altair HyperWorks®: HyperMesh® 1990-2015; HyperCrash® 2001-2015; OptiStruct® 1996-2015; RADIOSS®1986-2015; HyperView® 1999-2015; HyperView Player® 2001-2015; HyperStudy® 1999-2015; HyperGraph® 1995-2015; MotionView® 1993-2015; MotionSolve® 2002-2015; HyperForm® 1998-2015; HyperXtrude® 1999- 2015; Process Manager™ 2003-2015; Templex™ 1990-2015; TextView™ 1996-2015; MediaView™ 1999-2015; TableView™ 2013-2015; BatchMesher™ 2003-2015; HyperMath® 2007-2015; HyperWeld® 2009-2015; HyperMold® 2009-2015; Manufacturing Solutions™ 2005-2015; solidThinking® 1993-2015; solidThinking Inspire® 2009-2015; solidThinking Evolve® 1993-2015; Durability Director™ 2009-2015; Suspension Director™ 2009-2015; Acu-Solve® 1997-2015; AcuConsole® 2006-2015; SimLab® 2004-2015; Virtual Wind Tunnel™ 2012-2015; FEKO™ 1999-2015 and VisSim™ 1989-2015.

Altair Packaged Solution Offerings (PSOs) Copyright© 2008-2015 Automated Reporting Director™ 2008-2015; Impact Simulation Director™ 2010-2015; Model Mesher Director™ 2010-2015; Model Verification Director™ 2013-2015; and Squeak and Rattle Director™ 2012-2015.

Altair intellectual property rights are protected under U.S. and international laws and treaties. Additionally, Altair software is protected under patent #6,859,792 and other patents pending. All other marks are the property of their respective owners.ALTAIR ENGINEERING INC. Proprietary and Confidential. Contains Trade Secret Information. Not for use or disclosure

outside of Altair and its licensed clients. Information contained in Altair software shall not be decompiled, disassembled, “unlocked”, reverse translated, reverse engineered, or publicly displayed or publicly performed in any manner. Usage of the software is only as explicitly permitted in the end user software license agreement.

Copyright notice does not imply publication.

Third Party Licenses

Software Security Measures:

Altair Engineering Inc. and its subsidiaries and affiliates reserve the right to embed software security mechanisms in the Software for the purpose of detecting the installation and/or use of illegal copies of the Software. The Software may collect and transmit non-proprietary data about those illegal copies. Data collected will not include any customer data created by or used in connection with the Software and will not be provided to any third party, except as may be required by law or legal process or to enforce our rights with respect to the use of any illegal copies of the Software. By using the Software, each user consents to such detection and collection of data, as well as its transmission and use if an illegal copy of the Software is detected. No steps may be taken to avoid or detect the purpose of any such security mecha-nisms.

Technical Support

Location Telephone e-mail

North America +1 248 614 2425 [email protected]

China +86 (0)21 6117 1666 [email protected] +33 (0)1 4133 0992 [email protected]

Germany +49 (0)7031 6208 22 [email protected] +91 80 66 29 4500 [email protected]

Italy +39 800 905595 [email protected] +81 3 5396 2881 [email protected]

Korea +82 70 4050 9200 [email protected]

Scandinavia +46 (0) 46 460 2828 [email protected] +44 (0)1926 468 600 [email protected]

http://www.altairhyperworks.com/ClientCenter3rdPartyLicense.aspx

http://www.altairhyperworks.com/ClientCenter3rdPartyLicense.aspx

Table of Contents

Preface ix

1 About PBS Analytics™ 11.1 System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Supported Platforms and Browsers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.3 Prerequisites for installing and using PBSA . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.4 Chart rendering response time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2 Deployment Options 72.1 Deployment Option One . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.2 Deployment Option Two. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.3 Deployment Option Three. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3 Preparing to install PBSA 133.1 Analyzing each PBS Professional complex . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.2 Implementing PBS Professional changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.3 Installing PBSA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163.4 Configuring PBSA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.5 Parsing the accounting logs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

4 Migrating to PBSA 13.0 214.1 Migrating versions of PBSA prior to 12.3.x . . . . . . . . . . . . . . . . . . . . . . . . . . . 214.2 Migrating PBSA version 12.3.x and 12.4.x . . . . . . . . . . . . . . . . . . . . . . . . . . . 224.3 Preserving configuration files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

5 Installing in GUI mode 255.1 Installing on a single server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255.2 Installing on separate servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

6 Installing in console mode 656.1 Installing on a single server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 656.2 Installing on separate servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

PBS Analytics 13.0 Administrator’s Guide v

Table of Contents

7 Uninstalling PBSA 997.1 Uninstalling using the GUI mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 997.2 Uninstalling using the console mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

8 Post-installation steps 1058.1 Setting up the system path. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1068.2 Opening the PBSA service port. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1078.3 Resetting the database user password . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1078.4 Providing a license server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1088.5 Configuring PBSA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1088.6 Specify shared privileges. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1098.7 Starting PBSA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1098.8 Parsing the accounting logs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1108.9 Correcting recovery records . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1108.10 Analyzing unsupported records. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1128.11 Logging into the PBSA Web Application. . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

9 Supplying PBS node information 1139.1 Obtain node information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1149.2 Evaluate offline or downed nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1159.3 Run the node configuration command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1169.4 Node configuration for previous years . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

10 Supplying other site details 11910.1 Configuring holidays. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12110.2 Configuring prime and non-prime hours . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12210.3 Configuring custom groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12310.4 Configuring exit codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12410.5 Configuring custom resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

11 Starting PBSA 12711.1 Starting the PBSA service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12711.2 Starting the Data Collectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128

12 Parsing the accounting logs 13112.1 Tracking the parsing progress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13112.2 Tracking the caching and initialization progress. . . . . . . . . . . . . . . . . . . . . . . 13412.3 Initial parsing of the accounting logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13612.4 Daily parsing of the accounting logs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13612.5 Configuration changes after initial parsing . . . . . . . . . . . . . . . . . . . . . . . . . . . 137

13 Adding a PBS Complex 139

vi PBS Analytics 13.0 Administrator’s Guide

Table of Contents

14 PBSA service commands 14114.1 PBSA service commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14114.2 Data collector service commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

15 PBSA commands 14715.1 Analyzing a PBS Professional complex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14915.2 Configuring nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15415.3 Configuring holidays. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16415.4 Configuring custom groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17515.5 Configuring exit codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18315.6 Configuring the Data Collector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19015.7 Configuring the Parser. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19515.8 Resetting the PBSA dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20115.9 Changing the PBSA database password . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206

16 PBSA Measures & Dimensions 21116.1 Dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21216.2 Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223

17 Troubleshooting 22717.1 Log files and logging rollover policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22717.2 Troubleshooting the installation of PBSA. . . . . . . . . . . . . . . . . . . . . . . . . . . . 22817.3 Troubleshooting the PBSA service startup . . . . . . . . . . . . . . . . . . . . . . . . . . . 23117.4 Troubleshooting logging into PBSA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23317.5 Troubleshooting the Data Collector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23417.6 Troubleshooting licensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23617.7 Troubleshooting access to PBSA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23617.8 PBSA Log File Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237

18 PBSA configuration files 25518.1 Holiday configuration file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25518.2 Exit code configuration file. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26418.3 Custom group configuration file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26718.4 Custom resource configuration file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27418.5 Node details file. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275

PBS Analytics 13.0 Administrator’s Guide vii

Table of Contents

19 Advanced Configuration 30119.1 Disabling PBSA service ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30119.2 Configuring logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30419.3 Configuring the license server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30619.4 Configuring JVM Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31019.5 Configuring SSH Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31219.6 Changing the accounting log location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31319.7 Configuring communication between the DC and the PBSA service. . . . . . . 31419.8 Configuring plot point decimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31519.9 Changing the license option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31619.10 Configuring a site specific SSL certificate . . . . . . . . . . . . . . . . . . . . . . . . . . . 31719.11 Creating a site specific SSL keystore file . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31819.12 Changing the ports used by PBSA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319

20 Reparsing the accounting logs 323

21 Disaster Recovery 32521.1 Backing up PBSA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32521.2 Restoring PBSA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32821.3 Backing up the PBSA database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32921.4 Restoring the PBSA database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330

22 Peer Scheduling 333

23 PBSA Hooks 33523.1 Define the custom resource to PBSA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33723.2 Updating the node details configuration file . . . . . . . . . . . . . . . . . . . . . . . . . . 34323.3 Writing a converter class. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34423.4 Troubleshooting PBSA Hooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356

24 Using the PBSA Migration Tool 359

25 Glossary of Terms 361

Index 369

viii PBS Analytics 13.0 Administrator’s Guide

PrefaceThis document describes how to install, configure and administer PBSA.

Audience

This document is intended for administrators who perform the following tasks:

• Install and administer PBS Works products

• Monitor the operation of PBS Works products

• Troubleshoot PBS Works products

To use this document, you need to be familiar with PBS Professional concepts.

Documentation Accessibility

Documentation can be found at the following URL:

http://www.pbsworks.com/PBSProductGT.aspx?n=PBS-Analytics&c=Overview-and-Capa-bilities&d=PBS-Analytics,-Documentation

Related Documents

For more information, see the following PBSA documentation:

• PBS Analytics Frequently Asked Questions (FAQ)

• PBS Analytics Chart Calculations

• PBS Analytics User Guide

• PBS Analytics Release Notes

PBS Analytics 13.0 Administrator’s Guide ix

http://www.pbsworks.com/PBSProductGT.aspx?n=PBS-Analytics&c=Overview-and-Capabilities&d=PBS-Analytics,-Documentation

What’s New

Here’s a quick look at the latest features available with PBSA 13.0.

Improvements Lessen Installation and Configuration Time

The following improvements introduced with PBSA 13.0 make the installation quicker, and reduce the time necessary for post-installation configuration:

• During installation, the PBSA Service User automatically defaults to the user performing the installation, and the license server is automatically obtained from the Altair license path variable if present.

• The Data Collector copy time has been reduced from 5 minutes to 1 minute and the trans-portation time has been reduced from 10 minutes to 1 minute. Therefore, after installa-tion there is no need for configuration of the Data Collectors.

• The Parser delay has been reduced from 30 minutes to 15 minutes. For most sites this time interval is suitable for transporting the accounting logs to the machine where the Parser is installed. In cases where the delay must be configured see section 15.7, “Con-figuring the Parser”, on page 195.

Stability and Scalability

The following changes have been made to improve the stability and scalability of PBSA:

• The Parser runs as a separate leaner component. When problems arise, it is easier to restart the Parser without affecting other components.

• A reduced database schema size.

• The Mondrian server has been replaced with an optimized HPC domain aware query engine, improving query performance and database resource usage. It also improves chart building capabilities, such as sorting of x-axis values, selection of date range mini-mum and maximum values, and combining Time-Advanced Dimensions.

• The new query engine also reduces the dependency upon the node details and holiday files being tightly coupled with the product.

• Performance of parsing and caching improves linearly with an increase of cores.

Upgrades to Latest JDK

PBSA 13.0 uses Java 8 enabling improved security.

x PBS Analytics 13.0 Administrator’s Guide

Improvements to the Job Data Export Eliminates Manual Configuration

The PBSA 13.0 Web Application displays all relevant job data used to build a chart when a user clicks the Job Data button, or exports the job data to a comma separated values file. The systems administrator no longer needs to configure the values that will be displayed, therefore the “Configuring raw data fields” section has been removed from the PBSA13.0 Administra-tor’s Guide.

Ability to Reset PBSA Including All Data Collectors

The pbsa-data-reset command now automatically resets all Data Collectors making the process for discarding parsed accounting log data straightforward and uncomplicated.

Chart and Dashboards Sharing

A new category is available called “Shared”. PBSA users with special privileges can create, modify, and delete charts and dashboards within this category. All other users have viewing access to the charts and dashboards in the “Shared” category.

Charts and dashboards added to the “Shared” category are available to other users upon log-ging into PBSA, or by refreshing the browser’s cache (F5).

Optional Node Configuration Improves Installation Process

Prior to PBSA 13.0, node configuration was necessary before the PBS Professional account-ing logs could be parsed. Whenever a change was made to the node configuration, the previ-ously parsed data had to be discarded and the accounting logs had to be reparsed.

With PBSA 13.0, node configuration is no longer necessary, making installation quick and easy. System administrators are able to install PBSA 13.0 and then supply node information whenever they desire.

When PBS Professional node information is not supplied, the data displayed in the PBSA charts is confined to the cluster’s job activity (extracted from the PBS Professional accounting logs). Therefore, information such as how much memory and CPU is available and which nodes are down or unused is not available. Consequently, PBSA makes the following assumptions about these metrics:

• Availability = Used

• Downed nodes = 0

• Unused nodes = 0

• Percentage Utilization = 100%

Additionally, when node information is not supplied, the values for the PBSA Dimension Server are populated with the FQDN of the machines where the Data Collector is installed.

PBS Analytics 13.0 Administrator’s Guide xi

You may supply node information at any time and after the next Parser cycle completes, PBSA provides actual values for the above metrics, producing richer, more meaningful charts. The following table describes how the absence of node detail information will affect the default PBSA charts:

Table 2-1: Absence of node information and its affect on PBSA charts

Chart Name Effect of the Absence of Node Information

Jobs Count By Day No change to chart.

Jobs by Software by Group No change to chart.

Jobs by Software by Node Nodes having no job activity will not be represented in this chart.

Requested Vs Used Memory by Software

No change to chart.

Node Walltime by Group – Non Prime Time

Node CPU availability is not known, therefore the per-centage of node walltime used by a group will not take into account unused or underused CPU resources.

Node Walltime by Group –Prime Time

Node CPU availability is not known, therefore the per-centage of node walltime used by a group will not take into account unused or underused CPU resources.

Unused Node Capacity by Node

Node CPU availability is not known, therefore unused node capacity cannot be determined. It is assumed that all nodes were 100% utilized, thus unused node walltime is zero. Additionally, nodes having no job activity will not be represented in this chart.

Unused Node Capacity by Quarter

Node CPU availability is not known, therefore unused node capacity cannot be determined. It is assumed that all nodes were 100% utilized, thus unused node walltime is zero.

Used Vs Unused Node Cycle

Node CPU availability is not known, therefore unused node capacity cannot be determined. It is assumed that all nodes were 100% utilized, thus unused node walltime is zero. Additionally, nodes having no job activity will not be represented in this chart.

xii PBS Analytics 13.0 Administrator’s Guide

Jobs By Exit Status No change to chart.

Successful Vs Unsuccessful Jobs by Node

Node information is not available, therefore the chart will only display information about nodes where there was job activity. Nodes having no job activity will not be repre-sented in this chart.

Successful Vs Unsuccessful Jobs by Software

No change to chart.

User Job Efficiency and Productivity

No change to chart.

Wait Time by Software by User

No change to chart.

Wait Time VS Overall Time by Software

No change to chart.

Daily Percentage Core Utilization

Node CPU availability is not known, therefore unused core utilization cannot be determined. This chart will either show 100% utilization on days where there is job activity, or will show 0% utilization on days where there is no job activity.

Daily Resource Usage Node CPU availability is not known, therefore unused core utilization cannot be determined. It is assumed that all nodes were 100% utilized, thus unused cores is zero.

Queued Vs Running Memory by Software

No change to chart.

Running Cores Vs Queued Cores by Queue

No change to chart.

Running cores vs. Queued cores

Node CPU availability is not known, therefore it is assumed that available cores is equal to used cores.



PBS Analytics 13.0 Administrator’s Guide xiii

Reparsing Not Required for Holiday, Prime, and Non-Prime Time Changes

With previous versions of PBSA, configuration of the following site specific information was optional, and PBSA provided defaults for:

• holidays

• prime time hours

• non-prime time hours

However, once the PBS Professional accounting logs were parsed, changes to this information meant having to discard the PBSA data and reparse all accounting logs. Similar to the node configuration, PBSA 13.0 now allows this information to be changed at any time and the PBSA charts will reflect those changes after the next Parser cycle completed.

For more information about the configuration conditions that require reparsing of the PBS Professional accounting logs see section 12.5, “Configuration changes after initial parsing”, on page 137.

Revamped Custom Resource Hook

The custom resource configuration file has been converted to an XML format making the cus-tom resource definition less error prone. The file is now located at:

INSTALL_DIR_13.0/portal/binaries/parser/hooks

Additionally, the process for adding a custom resource to PBSA has been simplified, and error handling and logging has improved. For more information see Chapter 23, "PBSA Hooks", on page 335.

Weekly Percentage Core Utilization

Node CPU availability is not known, therefore unused core utilization cannot be determined. This chart will either show 100% utilization during weeks where there is job activity, or will show 0% utilization during weeks where there is no job activity.



xiv PBS Analytics 13.0 Administrator’s Guide

Improved Logging

Improved logging of error messages, including the reasons for discarding a job while parsing the PBS Professional accounting logs. The section 17.8, “PBSA Log File Messages”, on page 237 provides a list of messages with a description and any required action.

Events and errors that occur while parsing the node detail and holiday configuration files are now written to the PBS Works log file (pbsworks.log). For more information about the PBSA log files see section 17.1, “Log files and logging rollover policy”, on page 227.

Database Schema Changes Improve Parsing and Caching Times

Changes to the PBSA database schema have improved the time required to parse and cache the PBS Professional account log data. Refer to section 1.1.2, “Parsing benchmarking data”, on page 3 for benchmarking data based on licensing option. Due to these improvements, the custom query file is no longer required to cache additional queries, and the “Creating custom queries” section has been removed from the PBSA13.0 Administrator’s Guide.

Improved Job Counting

PBSA 12.4 displayed job count against states (finished and running) based on the state of the job as of now (when the chart was generated). Based on customer feedback, improvements have been made to the way in which PBSA counts jobs. PBSA 13.0 displays job counts against states per a particular day.

The following tables describes the differences between PBSA 12.4 and 13.0:

Table 2-2: Counting Jobs Comparison between PBSA 12.4 and 13.0

Job Jan1 Jan 2 Jan 3 Jan 4 Jan 5

1

2

3

4

5

PBSA 12.4 3 Finished 4 Finished 4 Finished 3 Finished 2 Finished

PBSA 13.0 3 Running

1 Finished

4 Running 4 Running

1 Finished

3 Running

1 Finished

2 Running

2 Finished

PBS Analytics 13.0 Administrator’s Guide xv

Security Bugs Fixed

The Poodle, Freak, and Logjam https security bugs are fixed.

New Parsing and Caching Progress Indicator Screen

The Web Application has a new Progress Indicator Screen to visualize the progression of the parsing and caching process.

Hide Zero Values Functionality Corrected

The Web Application functionality to hide chart values equal to zero has been fixed.

Unused Reporting Improvements

Based on customer feedback, changes have been made to PBSA so that unused resources are not reported by default. PBSA users have the ability to report upon unused usage by adding the Resource State -> Unused filter to a chart.

xvi PBS Analytics 13.0 Administrator’s Guide

Chapter 2

Conventions

Various conventions are used to help you more quickly identify special terms. The following table describes those conventions and provides examples of their use.

Table 2-3: Typographic conventions

Convention Meaning Example

Bold Bold typeface indi-cates emphasis or UI components.

This option CANNOT be used if there are multiple PBS Professional com-plexes.

Open a chart and then click the Job Data button on the Chart Title Bar.

Courier new Courier new typeface indicates names of com-mands, filepaths, and file-names.

Use the pbsa-setup-counsellor command to analyze each PBS Profes-sional complex.

The Data Collector will log messages to the file INSTALL_DIR_13.0/dc/logs/dc.log.

Italics Italic typeface indicates book titles or the argu-ment to a command option.

PBS exit codes can be found in the PBS Professional Administrator’s Guide.

pbsa-config-holidays --start-date=STARTDATE

Bold/Italics Bold/Italic typeface indi-cates a UI action.

Click Next.

[ ] Brackets enclose one or more command options. Do not enter the brackets.

[options]

{ } Braces enclose two or more items, one of which is required. Do not enter the braces.

{true | false}

| A vertical bar represents a choice of two or more options within brackets or braces. Enter one of the options. Do not enter the vertical bar.

{true | false}

PBS Analytics 13.0 Administrator’s Guide xvii

Chapter 2

PBS_HOME This represents the PBS Professional home direc-tory.

It defaults to PBS_HOME/server_priv/accounting.

PBS_EXEC This represents the PBS Professional execution directory.

The value for PBS_EXEC could not be found.

INSTALL_DIR_13.0 This represents the direc-tory where PBSA 13.0 is installed. The default installation location for PBSA is /opt/altair/pbsa/13.0

INSTALL_DIR_13.0/portal/services

INSTALL_DIR_12.x This represents the direc-tory where PBSA 12.x is installed.

INSTALL_DIR_12.x/portal/services

Table 2-3: Typographic conventions

Convention Meaning Example

xviii PBS Analytics 13.0 Administrator’s Guide

Chapter 1

About PBS Analytics™

Altair PBS Analytics™ (PBSA) is a graphical web-based accounting, analytics and reporting solution. Historical usage data can be queried with respect to jobs, applications, users, proj-ects, groups, and many more. The application provides PBS Works™ administrators with advanced job and license analysis to support data-driven planning and decision-making.

Companies and institutions of all sizes use high performance computing to develop innova-tive new products and create entirely new markets. As a result, application workloads have become much larger and more complex than ever before. Comprehensive and robust analysis and visualization features are needed to help ensure limited computing resources are being used effectively and efficiently. As a core part of a PBS Works™ solution, PBS Analytics turns oceans of data into insight, knowledge and intelligence.

With PBS Analytics users can:

• Access and view charts and graphs via a web browser interface, anytime and anywhere.

• Validate project planning assumptions and easily balance workloads to meet changing priorities.

• Generate detailed billing details to support chargeback accounting.

• Easily build customized performance dashboards for users, administrators and manage-ment.

• Dynamically create and graph historic usage data from one or many PBS Professional clusters.

PBS Analytics 13.0 Administrator’s Guide 1

Chapter 1 About PBS Analytics™

1.1 System Requirements

1.1.1 Optimal hardware requirements for PBSA engine

Notes:

• Two (2) GB of free RAM and ten (10) GB of free hard disk space are recommended for each instance of a Data Collector.

• Swap space is recommended in a separate physical drive.

Table 1-1: Optimal hardware requirements per license option

Licensing recordsLicense

typeCPU

RAM(GB)

Swap(GB)

Disk Space(GB)

up to 0.5 million Base 4 core 16 32 100

0.5 million to 2 million Standard 4 core 16 64 200

2 million to 5 million Standard 4 core 32 128 400

5 million to 50 million Advanced Consult PM team

2 PBS Analytics 13.0 Administrator’s Guide

PB 3

About PBS Analytics™ Chapter 1

1.

Ea

1.

If least 10 GB of free hard disk space sho rge PBS Professional accounting log da isk space being used by the PBS Profes-sio

ory byal)

Parsing time (min)

Cachingtime (min)

Total time Parsing + Caching

(min)

8 14 17.15 31.15

4 29 88.95 117.95

6 32 16.09 48.09

S Analytics 13.0 Administrator’s Guide

1.2 Parsing benchmarking data

ch row indicates benchmarking based on a specific customer’s data:

1.3 Minimum hardware requirements for Data Collectors

the Data Collector is installed on a separate server from the Web Application and Parser, then at uld be available on each server where the Data Collectors are installed. For sites with a very la

taset (30-50 million job records), allow for free hard disk space which is larger than the current dnal accounting log directory.

Table 1-2: Parsing Benchmarking Data

LicenseType

No. of E Recs

No. of License

Recs

No. of Nodes

DB Size No. of CPUs

RAM (GB)

Memory Used byMonetdb

(GB)

MemUsedPort(GB

Base

CentOS 6.4

118,077 123,628 15 58 MB 4 16 .96 8.1

Standard

CentOS 6.5

1,078,313 4,353,201 2,387 647 MB 4 32 5.05 20.3

Advanced

SLES 11 SP2

6,018,368 6,251,861 263 2.4 GB 8 64 5.95 18.9


1.2 Supported Platforms and Browsers

PBSA is supported on the following platforms:

Supported Operating Systems

Linux 64 bit

• SLES 10 SP4, SLES 11 SP2 and SLES 11 SP3

• RHEL 5.4+ and 6.x

• Cent OS 5.5+ and 6.x

Supported Browsers

Windows

• Firefox 10.x ESR+ (Only Enterprise Service Release is supported. Please refer to https://www.mozilla.org/en-US/firefox/organizations/.

• Chrome 15-43

• Internet Explorer 9-11

Linux

• Firefox 10.x ESR+ (Only Enterprise Service Release is supported. Please refer to https://www.mozilla.org/en-US/firefox/organizations/.

Mac OS X

• Safari 6+ (limited testing)

1.3 Prerequisites for installing and using PBSA

The following are prerequisites for installing PBSA:

• system administrator privileges (root privileges)

• existence of the following user before beginning the installation:

- The PBSA service user - This user must be an existing user on the server where PBSA is being installed and will be used to register the PBSA services. The PBSA and database services will be started under this user.

• a SSH server installed and running for authentication

• adherence to the minimum system requirements (see section 1.1, “System Require-ments”, on page 2).


https://www.mozilla.org/en-US/firefox/organizations/




About PBS Analytics™ Chapter 1

The following are prerequisites for using PBSA:

• PBSA must be installed on a machine that is running Python 2.4 or greater.

• a LMX license server version 12.0.3 or greater is required to login to PBSA.

It is recommended that the PBSA Web Application and Parser be installed on a server that is either a dedicated web portal server (a server that may have other Altair products deployed such as Compute Manager or Display Manager), or a server dedicated solely to the PBSA application.

1.4 Chart rendering response time

The data associated with the charts that come with PBSA is analyzed and cached when the PBSA service is started, and when new accounting log data is parsed. In addition, data asso-ciated with any charts created by the user that exists when the PBSA service is started is also analyzed and cached. This caching improves the overall response time of the PBSA Web Application. However, when a user creates a chart using a dimension or measure combination that is has not yet been cached, a noticeable slow-down in response time may be experienced due to the time necessary for analysis, aggregation, and caching of data. Further queries against this same data should not experience any degradation in response time.

This strategy also ensures that the customer is paying computational cost for charts which are out-of-the-box, or created based on their priority only.


Chapter 2

Deployment OptionsThe PBSA application consists of three components:

• PBSA Data Collector

• PBSA Parser

• PBSA Web Application

The main responsibility of the PBSA Data Collector is to make the PBS Professional account-ing logs accessible to the PBSA Web Application. The Data Collector has two functions. It first copies the accounting logs from your site’s PBS Professional accounting log location to an intermediate holding area. Secondly, it transports the information stored in the holding area to the machine where the PBSA Web Application and Parser are installed.

The PBSA Parser reads the PBS Professional accounting logs, parses the appropriate informa-tion from the accounting logs, and then loads this information into a database.

The PBSA Web Application is a graphical web-based accounting, analytics and reporting application.

The PBSA Data Collector runs under its own service, while a second PBSA service is respon-sible for running the Parser, Web Application, and associated database. The service responsi-ble for the Data Collectors will be referred to as the PBSA Data Collector service, while the service responsible for the Parser and Web Application will be referred to as the PBSA ser-vice.

Due to the I/O intensive workload of PBSA, it is recommended that the PBSA Web Applica-tion and Parser be installed on a server that is either a dedicated web portal server (a server that may have other Altair products deployed such as Compute Manager or Display Man-ager), or a server dedicated solely to the PBSA application (see section 1.1, “System Require-ments”, on page 2).


Chapter 2 Deployment Options

Each PBSA Data Collector is dedicated to a PBS Professional complex, therefore a PBSA Data Collector must be installed for each PBS Professional complex.

If your site implements peer scheduling see Chapter 22, "Peer Scheduling", on page 333 for additional information on how to deploy and configure PBSA.

Three deployment options are available for PBSA:

• The PBSA Parser, Web Application and Data Collector are deployed on a single machine (see section 2.1, “Deployment Option One”, on page 9).

• The PBSA Web Application and Parser are deployed on a separate machine from the Data Collector (see section 2.2, “Deployment Option Two”, on page 10).

• Install PBSA with a demonstration database (see section 2.3, “Deployment Option Three”, on page 11).


Deployment Options Chapter 2

2.1 Deployment Option One

The first deployment option is recommended for sites where there is only one PBS Profes-sional complex. This option deploys the PBSA Web Application. Parser, and Data Collector on a single machine.

Requirements for this deployment option are:

• The PBSA Data Collector must have access to the PBS Professional accounting logs. For example, through a shared file system.

• This option cannot be used if there are multiple PBS Professional complexes.

• Installation of the PBSA Web Application, Parser, and Data Collector must be installed through a single installation process.

.

Installation of all three PBSA components onto a single machine must be done through a single installation process. The Data Col-lector cannot be installed separately.



2.2 Deployment Option Two

The second deployment option is recommended for sites where there are one to many PBS Professional complexes. This option deploys the PBSA Web Application and Parser on a sep-arate machine from the Data Collector.

Please note - if a site has PBS Professional complexes that reside in different time zones, PBSA will convert and report any time values (start time, end times, etc.) in the time zone of the machine where the PBSA Web Application and Parser are installed.

Requirements for this deployment option are:

• The PBSA Data Collector is deployed on each machine where the PBS Professional accounting logs are located (typically the PBS Professional head node).

• Communication between the PBSA Web Application and Data Collector is done over https, and is configured during installation of the PBSA components.

• Installation of the PBSA Web Application and Parser must be done separately from the Data Collectors, and a Data Collector must be installed for each PBS Professional com-plex


Deployment Options Chapter 2

2.3 Deployment Option Three

The third deployment option is recommended for sites where a quick installation of PBSA for evaluation purposes is the priority. This option does not install the PBSA Parser component, therefore access to PBS Professional accounting logs is not necessary. Instead, a fully loaded demonstration database is installed so that all out-of-the box charts, dashboards, and slide-shows can be viewed, and all key features of PBSA can be accessed and evaluated.

This option is similar to option one where the PBSA Web Application, Parser and Data Col-lector are deployed on a single machine. However, during installation you will direct PBSA to use the demonstration database, rather than providing the location to the PBS Professional accounting logs.

If you choose to install the demonstration database, then the PBSA Parser will not be installed.


Chapter 3

Preparing to install PBSABefore installing PBSA, review the following topics to understand the process for installing and configuring PBSA, and parsing the PBS Professional accounting logs.

1. Analyzing each PBS Professional complex.

2. Implementing PBS Professional changes to get the full benefit of reporting through the PBSA Web Application.

3. Installing PBSA including the Web Application, Parser and Data Collectors.

Special steps are necessary for sites that have a previous version of PBSA installed. For more information see Chapter 4, "Migrating to PBSA 13.0", on page 21.

4. Configuring PBSA to supply site specific information to improve PBSA’s reporting capabilities.

5. Parsing the accounting logs.

3.1 Analyzing each PBS Professional complex

Use the pbsa-setup-counsellor command to analyze each PBS Professional complex. This command will inspect the accounting logs for a PBS Professional complex and return information that is useful for installing and configuring PBSA.For more information about this command see section 15.1, “Analyzing a PBS Professional complex”, on page 149.

See the PBS Analytics 13.0 Release Notes for information about where you can download this command prior to installing PBSA. Once PBSA is installed, this command is available as part of the PBSA installation.


Chapter 3 Preparing to install PBSA

3.2 Implementing PBS Professional changes

Various default charts display data about node classes, applications, and projects. Several steps must be done within PBS Professional to realize the full benefit of this reporting:

• Classifying the PBS Professional nodes

• Utilizing the PBS resource “software”

• Utilizing the qsub -P option

3.2.1 Classifying the PBS Professional nodes

PBSA can provide utilization metrics for groups or classifications of PBS Professional nodes, but to obtain these metrics a PBS Professional node must be assigned or classified to a nodeclass. This is accomplished by creating a PBS Professional custom resource called “nodeclass” on each PBS Professional node.

You can classify your PBS Professional nodes in any way that you choose for your site’s installation. For example, your site may have purchased HP hardware in 2008, and then pur-chased IBM hardware in 2012, so you may want to classify your nodes by hardware configu-ration. A nodeclass called “HP” could be defined for each node within the HP hardware configuration. A second nodeclass could be defined called “IBM” for each node in the second hardware configuration. PBSA can then provide utilization metrics based on these hardware configurations.

Another example is to classify nodes by discipline. Some sites may assign certain disciplines to specific nodes, such as CFD is assigned to node A, while CAD is assigned to node B. PBSA can provide utilization metrics for disciplines by defining a nodeclass of “CFD” on node A, and defining a nodeclass of “CAD” on node B.

For more information on how to define a PBS Professional custom resource see the PBS Pro-fessional Administrator’s Guide - PBS Resources - Custom Resources.

Prior to parsing the PBS accounting logs, create a custom resource on each PBS Professional node called “nodeclass” so that PBSA can provide utilization statistics for groups or classifications of PBS Professional nodes.


Preparing to install PBSA Chapter 3

3.2.2 Utilizing the PBS resource “software”

PBSA can provide metrics at the application level, if the PBS Professional built-in resource “software” is utilized. For a description of the built-in resource “software” see the PBS Pro-fessional Reference Guide - Resources - Built-in Resources. Using the built-in resource “soft-ware” allows PBSA to track how often a particular solver is utilized, for instance PBSJobApp.

The “software” built-in resource is requested as part of the qsub command. To request a value for software, add this to the qsub command:

-l software=<requested software>

or add this to the job script:

#PBS -l software=<requested software>

For more information see the PBS Professional User’s Guide - Submitting a PBS Job - Requesting Resources.

If you are using Compute Manager to submit jobs, then the “software” built-in resource is automatically set, therefore no special steps are necessary.

3.2.3 Utilizing the qsub -P option

PBSA can provide metrics at a project level, if the -P option is utilized when submitting a job (introduced in the 11.2 release of PBS Professional). It is highly recommended that all job submissions use this option for optimal utilization reporting. For more information see the PBS Professional User’s Guide - Submitting a PBS Job - Specifying a Job’s Project. Using the -P option allows PBSA to track utilization at a project level.

To submit a job and set the project, add this to the qsub command:

qsub .... -P <project name>

or add this to the job script:

#PBS ..... -P <project name>

If your site is using PBS Application Services 12.1or later, you can specify the job’s project using an application definition. See the recipe “How to Set the Job Project” in PBS Applica-tion Services Diving Into Application Definitions.

Use the PBS Professional “software” built-in-resource to track solver utilization.



If your site is using a version of PBS Professional prior to 11.2, PBSA provides an alternative to track utilization at the project level. Please refer to section 15.4, “Configuring custom groups”, on page 175 for more information. Keep in mind that this alternative has been made available for legacy purposes only, and may be deprecated in future releases of PBSA. The use of the qsub -P option is highly recommended as the preferred method of tracking utili-zation at a project level.

3.3 Installing PBSA

Installation of PBSA is similar to the installation of other PBS Works products. A binary or executable is downloaded from the PBS Works user area and installed on the appropriate Linux servers.

The following information is useful prior to installing PBSA:

• Options for loading the PBSA database

• Authentication options

• Default installation directory

• Licensing options

• Ports used by PBSA

3.3.1 Options for loading the PBSA database

During installation you will have a choice between installing a fully loaded demonstration database, or using your site’s accounting logs to load the PBSA database.

The demonstration database is useful when a quick installation for evaluation of the default charts and dashboards available with the PBSA Web Application is the priority.

Use the qsub -P option so that PBSA can provide metrics at a proj-ect level.

When you choose the demonstration database option, the PBSA Parser component is not installed, and post-installation configura-tion and parsing are not necessary.



If you choose to use your site’s accounting logs, then post-installation configuration and pars-ing of the accounting logs must be completed before the PBSA service can be started and available for use.

3.3.2 Authentication options

PBSA uses SSH to verify credentials of users who login.

3.3.3 Default installation directory

The default installation directory is:

/opt/altair/pbsa/13.0

3.3.4 Licensing options

During installation of PBSA, there are three licensing options (HWU is Hyper Works Unit):

• Base - up to 500,000 licensing records at 10 HWU + 1 HWU per user

• Standard - up to 5 million licensing records at 30 HWU + 1 HWU per user

• Advanced - up to 50 million licensing records at 60 HWU + 1 HWU per user

Licensing is dependent on the number of “licensing records”. A “licensing record” is not equivalent to a PBS Professional job accounting log "E" record. There is a “licensing record” for every “job node”. For example:

• A single job that runs on 100 nodes equates to 100 records for licensing.

• Ten jobs that ran on 1 node equates to 10 records for licensing.

• Ten jobs that each use 4 nodes equates to 40 licensing records.

• A single job that runs on 20 nodes for 1 hour equates to 20 licensing records.

• A single job that runs on 4 virtual nodes equates to 4 licensing records.

If a site adds additional data to the PBS Analytics database (not related to jobs), that data is also counted as far as licensing is concerned. For example, if a site adds "node temperature" once per day on 256 nodes, that would add 256 * 365 = 93,440 license records per year. PBSA will parse all accounting log data, regardless of the chosen licensing option. If the amount of licensing records exceeds the licensing option, then the PBSA Web Application will display the most recent accounting log data up to the amount of data that it is licensed for, effectively filtering out any data that exceeds the licensing limit (older data). For instance, if a licensing option of “base” is selected during installation, then PBSA will display the most recent PBS Professional accounting log data up to 500,000 licensing records.



Over time the licensing option may need to be changed to accommodate an increase in the number of accounting log job records. Upgrading the licensing option is a manual process. See section 19.9, “Changing the license option”, on page 316 for instructions. Upon upgrad-ing the license option, all data will become visible, unless the accounting log activity exceeds the newest licensing option.

3.3.5 Ports used by PBSA

During the installation of PBSA, the following ports will be configured:

• http port for the PBSA service = 9000

• https port for the PBSA service = 9143

• https port for the PBSA Data Collector service = 9343

• port for the PBSA database = 9200

The installer will try to configure each of these default ports. If a port is not available during installation, then PBSA will increment the port number by one and try again. This will con-tinue until a free port is found, or until 100 consecutive ports are refused at which point the installer will ask for a port.

A default SSL certificate for the PBSA service https connection will be configured automati-cally by PBSA. When accessing the Web Application over https, a user will see a browser warning about a security certificate that is not trusted. The user must accept the certificate to proceed to the PBSA login screen.

If you would like to use your own SSL certificate, rather than the default certificate issued by PBSA, see section 19.10, “Configuring a site specific SSL certificate”, on page 317.

Communication between the PBSA service and Data Collector service will be done exclu-sively over https.

If you do not want users to access the PBSA Web Application through both the http and https ports, see section 19.1, “Disabling PBSA service ports”, on page 301 for information on dis-abling either port.

PBSA will parse all accounting log data, regardless of the chosen licensing option, but will only display the most recent accounting log data up to the amount of data that it is licensed for, effectively filtering out any data that exceeds the licensing limit.



Once installation is complete, the PBSA service port must be opened by a site’s system administrator for incoming TCP traffic to allow communication between the PBSA service and the Data Collector service.

3.4 Configuring PBSA

Providing site specific information about your HPC cluster improves PBSA’s ability to col-lect, analyze and report upon job activity. With PBSA 13.0, configuration is optional, how-ever you may supply the following site specific information to improve reporting:

• PBS Professional nodes

• holidays

• prime and non-prime time hours

• exit codes

• custom groups

• PBS Professional custom resources

When PBS Professional node information is not supplied, the data displayed in the PBSA charts is confined to the cluster’s job activity (extracted from the PBS Professional accounting logs). You may supply node information at any time and after the next Parser cycle completes, PBSA provides actual values for the above metrics, producing richer, more meaningful charts. For a more complete explanation see “Optional Node Configuration Improves Installation Process” on page xi.

PBSA provides defaults for holidays, prime/non-prime time hours, and custom groups for the last five years. It also supplies defaults for exit codes. These defaults are summarized in Table 10-1, “Configuration defaults,” on page 120. You may use these defaults or provide site specific information.

Similar to the node configuration, PBSA 13.0 now allows holiday, prime time, and non-prime time information to be changed at any time and once the next Parser cycle completes, the PBSA charts will reflect those changes without having to reparse the accounting logs. For more information see “Reparsing Not Required for Holiday, Prime, and Non-Prime Time Changes” on page xiv.

Please note the port assigned to the PBSA service and Data Collec-tor service during installation. The port information is displayed during the “Pre-Installation Summary” and “Installation Complete” steps of the installation process.



3.5 Parsing the accounting logs

3.5.1 Parsing the accounting logs for the first time

Parsing of the accounting logs is necessary to make PBS Professional job data available to the PBSA Web Application. The Parser reads the PBS Professional accounting logs, parses the appropriate information from the accounting logs, and then loads this information into a data-base used by the Web Application.

The initial parsing process may take several hours depending upon the amount of data stored in the PBS Professional accounting log(s), number of processors, and amount of available RAM.

Once the initial parsing cycle is complete, changes to the following PBSA configuration files will require any parsed data to be discarded, and all accounting log data to be reparsed:

• exit codes

• custom groups


3.5.2 Daily parsing of the accounting logs

Once PBSA is configured properly and the PBS Professional accounting logs have been ini-tially parsed, the parsing cycle runs on a daily basis parsing the newly generated accounting logs.


Chapter 4

Migrating to PBSA 13.0

4.1 Migrating versions of PBSA prior to 12.3.x

Due to major changes to chart definitions introduced in PBSA 12.4, user charts, dashboards, and slideshows created with versions of PBSA prior to 12.3.x cannot be preserved and used by PBSA 13.0. They must be recreated once PBSA 13.0 is installed.

In addition, the following PBSA 12.x configuration files must be recreated:

• all holiday_YYYY.conf files

• all custom_group_YYYY.conf files

• exit_status.conf file

The following steps describe the process for migrating from versions of PBSA prior to 12.3.x to PBSA 13.0:

1. Uninstall PBSA 12.x. See the chapter “Uninstalling PBSA” in the PBS Analytics 12.x Administrator’s Guide for more information.

2. Install PBSA 13.0. For more information see Chapter 5, "Installing in GUI mode", on page 25 or Chapter 6, "Installing in console mode", on page 65.

3. Optionally configure PBSA 13.0 and reparse all PBS Professional accounting logs. For more information see Chapter 8, "Post-installation steps", on page 105.

4. Recreate all user charts, dashboards, and slideshows.


Chapter 4 Migrating to PBSA 13.0

4.2 Migrating PBSA version 12.3.x and 12.4.x

Migrating involves installing PBSA 13.0 on the same machine as PBSA 12.3.x or PBSA 12.4.x, and then running a migration tool installed with PBSA 13.0.

The following are steps for migrating versions of PBSA 12.3.x or 12.4.x to PBSA 13.0:

1. Preserve the 12.3.x or 12.4.x configuration files.

Before installing PBSA 13.0, you may preserve configuration files that are currently in use by your 12.3.x or 12.4.x installation. These files can be used after installing PBSA 13.0. For more information see section 4.3, “Preserving configuration files”, on page 22.

2. Migrate the 12.3.x or 12.4.x user charts, dashboards, and slideshows to a 13.0 format. Steps for this migration can be found in Chapter 24, "Using the PBSA Migration Tool", on page 359.

3. Uninstall PBSA 12.3.x or 12.4.x.

4. Import the preserved configuration files.

Import the previously preserved files into PBSA 13.0 by copying the PBSA configu-ration backup files to the following directory:

INSTALL_DIR_13.0/portal/services/pbsa/parser/modules/pbsdata/config

5. Optionally, configure PBSA 13.0 and reparse all PBS Professional accounting logs. For more information see Chapter 8, "Post-installation steps", on page 105.

4.3 Preserving configuration files

Before installing PBSA 13.0, you may preserve configuration files that are currently in use by your PBSA 12.3.x or 12.4.x installation. These files can be used after installing PBSA 13.0:

• all holiday_YYYY.conf files

• all custom_group_YYYY.conf files

• exit_status.conf file

Copy these files from the following location to a backup directory:

INSTALL_DIR_12.x/portal/services/pbsa/parser/modules/pbsdata/config

The backup directory should not be located within the PBSA 12.3.x or 12.4.x installation directory, as the contents of the installation directory will be removed during the uninstalla-tion process.


Migrating to PBSA 13.0 Chapter 4

If you do not preserve these files, they will have to be recreated using the PBSA Commands.

The node details configuration files (node_details_YYYY.conf files) cannot be pre-served, but can be regenerated using the pbsa-config-nodes command.


Chapter 4 Migrating to PBSA 13.0


Chapter 5

Installing in GUI modeSpecial steps are necessary for sites that have a previous version of PBSA installed. For more information see Chapter 4, "Migrating to PBSA 13.0", on page 21.

Prior to installing PBSA, it is recommended to use the pbsa-setup-counsellor com-mand to analyze each PBS Professional complex. This command will provide information necessary for the installation of PBSA, such as the number of licensing records found in the PBS Professional accounting log, and approximate disk space requirements.

Installing PBSA is quick and easy. The GUI based installer walks you through the process, requesting installation and configuration information along the way. Once installation is com-plete, there are post-installation tasks that need to be completed before PBSA is ready for use (see Chapter 8, "Post-installation steps", on page 105).

5.1 Installing on a single server

This option is recommended for:

• Deployment Option 1 - A site that has only one PBS Professional complex.

• Deployment Option 3 - A site that has chosen to configure PBSA to use the demonstra-tion database.

This option CANNOT be used if there are multiple PBS Professional complexes.

If you are installing the demonstration database, the PBSA Parser will not be installed.


Chapter 5 Installing in GUI mode

Prerequisites for installation of this product are:

• system administrator privileges (root privileges).

• a PBSA service user. This user must be an existing user on the server where PBSA is being installed and will be used to register the PBSA services. The PBSA and database services will be started under this user.

• a SSH server installed and running.


5.1.1 Download the installer

To install PBSA, download the installer appropriate for your platform from the user area.

5.1.2 Starting the installer

Start the installer by executing the following command:

./PBSA_13.0_Linux_x86_64.bin -i swing


It is recommended that the PBSA Web Application and Parser be installed on a server that is either a dedicated web portal server (a server that may have other Altair products deployed such as Com-pute Manager or Display Manager), or a server dedicated solely to the PBSA application.


Installing in GUI mode Chapter 5

5.1.3 Splash screen

After extracting the files, the installation will proceed by displaying the splash screen.



5.1.4 Introduction

An Introduction dialog box will be displayed. Please read through the instructions before pro-ceeding with the installation. To ensure a successful installation, please note the prerequisites for the installation. At anytime during the installation process, you may click Previous to go back to a previous installation step. To cancel the installation process, click Cancel. To con-tinue the installation process, click Next.



5.1.5 License Agreement

License agreement information will be displayed. Please read through the License Agreement before proceeding with the installation. To accept the terms of the license agreement, select I accept the terms of the License Agreement, and click Next.



5.1.6 Install options

If you are installing the demonstration database, select the following component:

• Analytics Application and Parser

Otherwise, if you are installing the Web Application, Parser and Data Collector on a single server (see section 2.1, “Deployment Option One”, on page 9), select both components:


• Data collector

Click Next.

If the demonstration database is chosen only the Web Application will be installed. PBSA will automatically forgo the installation of the Parser since accounting logs are not accessible with this option.

The Data Collector cannot be installed after installing the Web Application/Parser. Installation of all three components must be done through a single installation process.



5.1.7 Dataset

Select whether to configure PBSA to use a demonstration database that is already loaded with test data, or to configure PBSA to use your site’s accounting logs by selecting the appropriate choice.

The demonstration database is handy for quick installations where a demonstration of default charts is the priority.

If you choose to use your accounting logs, then you will be asked to specify a path to your PBS Professional accounting logs.

Make your selection and then click Next.

If you choose to install PBSA with the demonstration database, the PBSA Parser component will not be installed.



5.1.8 Accounting logs location

If you have chosen to configure PBSA to use your site’s PBS Professional accounting logs, then you will be prompted to specify the location of the accounting logs. The accounting logs can be available locally or via a mounted path on a shared file system. To restore the default location, click Restore Default. Once you have entered the location of the accounting logs, click Next.



5.1.9 Authentication server

PBSA uses SSH to verify credentials of users who login. You can either specify the SSH server running on the local machine, or specify a different host running SSH. Click Next to proceed with the default authentication server, or specify the SSH server hostname, and click Next.



5.1.10 PBSA service user

Enter a user id that will be used to run the PBSA services. This user must be a pre-existing and preferably non-privileged user on the server where PBSA is being installed. Click Next.

See section 17.2, “Troubleshooting the installation of PBSA”, on page 228 for any problems experienced while installing PBSA.

The PBSA and database services will run under the service user.



5.1.11 Installation location

Specify a location where the selected PBSA components will be installed. These components can be installed to the default location, another location locally, or a mounted location on a shared file system. To select a custom directory click Choose. To restore the default direc-tory, click Restore Default Folder. The default installation directory is:


Once an installation location is specified, click Next.



5.1.12 License server

Choose the licensing option you want to use (see section 3.3.4, “Licensing options”, on page 17 for more information). The default is base (HWU is Hyper Works Unit).




Click Next after making your choice.


It is not required to provide the license server at installation time. You will however be required to configure the license server infor-mation prior to starting the PBSA service (see section 19.3.1, “Ini-tial license server configuration”, on page 306).



5.1.13 Pre-installation summary

As a last step before installation a pre-installation summary will be displayed. If the listed specifications are incorrect, use the Previous button to go back and make any necessary changes. If all specifications are correct then proceed with the PBSA installation using the Install button.

IMPORTANT - Please note the port number of the PBSA service. This information will be needed for opening the port for incoming TCP traffic.



5.1.14 Installation progress

You can monitor the progress of your installation via the installation progress bar.



5.1.15 Start PBSA services

If you chose to install the demonstration database, then during the installation process a mes-sage will be displayed asking if the PBSA services should be started.

If a license server was not configured for PBSA during installation, do no start the PBSA ser-vice. You will need to configure the license server prior to starting the PBSA service.

Otherwise, select “Yes” and click Next to start the PBSA service.

If the PBSA service is not started, then it must be manually started once the installation pro-cess is complete - see section 14.1.1, “Starting the PBSA service”, on page 141.

See section 17.3, “Troubleshooting the PBSA service startup”, on page 231 for any problems experienced during PBSA service start-up.



5.1.16 Confirm PBSA services should not be started

If you choose to defer the startup of the PBSA service, you will be asked to confirm your choice. Click OK to confirm this action. .



5.1.17 Installation complete

Once the installation of PBSA is complete the installation details will be displayed. Click Done to exit the installation.



5.1.18 Post-installation

5.1.18.1 PBSA is configured to use the demonstration database

If you have configured PBSA to use the demonstration database, and did not provide a license server during installation, then you must now configure PBSA with a valid license server before proceeding (see section 19.3.1, “Initial license server configuration”, on page 306).

Once a valid license server is known to PBSA, the Web Application is available for use. Use any of the supported browsers (see section 1.2, “Supported Platforms and Browsers”, on page 4) to login to PBSA through the URL http://<hostname>:9000/pbsworks or https://<hostname>:9143/pbsworks.

PBSA may use a non-default port, if your site is already using the above ports for another pur-pose. Please contact your systems administrator if you cannot connect to PBSA through either of these URL's.

5.1.18.2 PBSA is configured to use the PBS accounting logs

If you have configured PBSA to use the PBS Professional accounting logs, please complete the post-installation activities (see Chapter 8, "Post-installation steps", on page 105) before using the PBSA Web Application.

5.2 Installing on separate servers

This deployment option is required for sites which want to analyze more than one PBS Pro-fessional complex:

• The PBSA Web Application and Parser are installed on a separate machine from the Data Collectors.

• A PBSA Data Collector must be installed either on the head node of each PBS Profes-sional complex, or on a machine that has access to the PBS Professional accounting logs for that complex.





• adherence to the minimum system requirements (see section 1.1, “System Require-



ments”, on page 2).


To install PBSA download the installer appropriate for your platform from the user area.

5.2.2 Installing the PBSA Web Application and Parser

5.2.2.1 Starting the installer



5.2.2.2 Splash screen





5.2.2.3 Introduction

An Introduction dialog box will be displayed. Please read through the instructions before pro-ceeding with the installation. To ensure a successful installation, please note the prerequisites for the installation. At anytime during the installation process, you may click Previous to go back to a previous installation step. To cancel the installation process, click Cancel. To con-tinue the installation process, click Next.



5.2.2.4 License Agreement




5.2.2.5 Install options

Verify that only the PBSA Web Application and Parser component is selected. Click Next.

5.2.2.6 Confirm PBSA Web Application and Parser install

A message will be displayed stating that you are installing only the PBSA Web Application and Parser. Click OK.

Only the PBSA Web Application and Parser will be installed.



5.2.2.7 Dataset

Configure PBSA to use your site’s accounting logs. Do not select the demonstration database option. Click Next.



5.2.2.8 Authentication server

PBSA uses SSH to verify credentials of users who login. You can either specify the SSH server running on the local machine, or specify a different host running SSH. Click Next to proceed with the default authentication server, or specify the SSH server hostname, and click Next.



5.2.2.9 PBSA service user

Enter a user id that will be used to run the PBSA services. This user must be a pre-existing and preferably non-privileged user on the server where PBSA is being installed. Click Next.





5.2.2.10 Installation location

Specify a location where the selected PBSA components will be installed. These components can be installed to the default location, another location locally, or a mounted location on a shared file system. To select a custom directory click Choose. To restore the default direc-tory, click Restore Default Folder. The default installation directory is:


Once an installation location is specified, click Next.



5.2.2.11 License server

Choose the licensing option you want to use (see section 3.3.4, “Licensing options”, on page 17 for more information). The default is base (HWU is Hyper Works Units).




Click Next.





5.2.2.12 Pre-installation summary


IMPORTANT - Please note the port number of the PBSA service. This information will be needed during the installation of the PBSA Data Collectors, and for opening the PBSA service port for incom-ing TCP traffic.



5.2.2.13 Installation progress




5.2.2.14 Installation complete

Once the installation of the PBSA Web Application and Parser is complete, the installation details will be displayed.

Now the installation of the PBSA Data Collectors on each of the PBS Professional complexes must be completed (see section 5.2.3, “Installing the PBSA Data Collector”, on page 55).

Click Done.



5.2.3 Installing the PBSA Data Collector

These installation steps must be repeated for each PBS Professional complex. A PBSA Data Collector must be installed either on the head node of each PBS Professional complex, or on a machine that has access to the PBS Professional accounting logs for that complex.




5.2.3.2 Splash screen





An Introduction dialog box will be displayed. Please read through the instructions before pro-ceeding with the installation. To ensure a successful installation, please note the prerequisites for the installation. At anytime during the installation process, you may click Previous to go back to a previous installation step. To cancel the installation process, click Cancel. To con-tinue the installation process, click Next .



5.2.3.4 License Agreement





Verify that only the Data Collector component is selected. Click Next.

5.2.3.6 Confirm PBSA Data Collector install

A message will be displayed stating that you are installing only the PBSA Data Collector. Click OK.

Only the PBSA Data Collector will be installed.



5.2.3.7 PBSA Web Application and Parser machine location

Specify the hostname and port number of the machine where the PBSA Web Application and Parser were installed. Click Next.



5.2.3.8 Accounting logs location

Specify the location of the PBS Professional accounting logs. The accounting logs can be available locally or via a mounted path on a shared file system. To restore the default loca-tion, click Restore Default. Once you have entered the location of the accounting logs, click Next.

.




You may install PBSA in the default directory, or you may select a custom directory by click-ing Choose. To restore the default directory, click Restore Default Folder. The default installation directory is:


Once an installation location is entered, click Next.




Once the installation of the PBSA Data Collector is complete the installation details will be displayed. Click Done to exit the installation.

Repeat this installation process for each PBS Professional complex.

After Data Collectors are installed on every PBS Professional complex, perform the post installation activities (see Chapter 8, "Post-installation steps", on page 105) to configure PBSA and make it ready for use.


Chapter 6

Installing in console modeSpecial steps are necessary for sites that have a previous version of PBSA installed. For more information see Chapter 4, "Migrating to PBSA 13.0", on page 21.

Prior to installing PBSA, it is recommended to use the pbsa-setup-counsellor com-mand to analyze each PBS Professional complex. This command will provide information necessary for the installation of PBSA, such as the number of licensing records found in the PBS Professional accounting log, and approximate disk space requirements.

Installing PBSA is quick and easy. The console based installer walks you through the process, requesting installation and configuration information along the way. Once installation is com-plete, there are post-installation tasks that need to be completed before PBSA is ready for use (see Chapter 8, "Post-installation steps", on page 105).

6.1 Installing on a single server

This option is recommended for:

• Deployment Option 1 - A site that has only one PBS Professional complex.

• Deployment Option 3 - A site that has chosen to configure PBSA to use the demonstra-tion database.

This option CANNOT be used if there are multiple PBS Professional complexes.

If you are installing the demonstration database, the PBSA Parser will not be installed.


Chapter 6 Installing in console mode







To install PBSA, download the installer appropriate for your platform from the user area.

6.1.2 Starting the installer


./PBSA_13.0_Linux_x86_64.bin -i console




Installing in console mode Chapter 6

6.1.3 Introduction

The console installer will launch. To ensure a successful installation, please note the prerequi-sites for the installation..

Press <ENTER> when you are ready to initiate the installation. You may cancel the installa-tion process at anytime by entering "quit" and pressing <ENTER>. At anytime during the installation process, you may enter "back" and press <ENTER> to go back to a previous installation step.



6.1.4 License agreement

Please page through the License Agreement by pressing <ENTER> until you are prompted to accept the License Agreement. To accept the License Agreement terms and conditions and continue with the installation process, enter "Y" and press <ENTER>.



6.1.5 Install options

If you are installing the demonstration database, select the following component:


by entering “1” and pressing <ENTER>.

Otherwise, if you are installing the Web Application, Parser and Data Collector on a single server (see section 2.1, “Deployment Option One”, on page 9), select both components by pressing <ENTER>:

If the demonstration database is chosen only the Web Application will be installed. PBSA will automatically forgo the installation of the Parser since accounting logs are not accessible with this option.

The Data Collector cannot be installed after installing the Web Application/Parser. Installation of all three components must be done through a single installation process.



6.1.6 Dataset

Select whether to configure PBSA to use a demonstration database that is already loaded with test data, or to configure PBSA to use your site’s accounting logs by selecting the appropriate choice.

The demonstration database is handy for quick installations where a demonstration of default charts is the priority.

If you choose to use your accounting logs, then you will be asked to specify a path to your PBS Professional accounting logs.

Make your selection and then press <ENTER>.

6.1.7 Accounting logs location

If you have chosen to configure PBSA to use your site’s PBS Professional accounting logs, then you will be prompted to specify the location of the accounting logs. The accounting logs can be available locally or via a mounted path on a shared file system.

If you choose to install PBSA with the demonstration database, the PBSA Parser component will not be installed.



6.1.8 Authentication server

PBSA uses SSH to verify credentials of users who login. You can either specify the SSH server running on the local machine, or specify a different host running SSH. Press <ENTER> to proceed with the default authentication server, or specify the SSH server host-name, and press <ENTER>.



6.1.9 PBSA service user

Enter a user id that will be used to run the PBSA services. This user must be a pre-existing and preferably non-privileged user on the server where PBSA is being installed. Press <ENTER>.





6.1.10 Installation location

Specify a location where the selected PBSA components will be installed. These components can be installed to the default location, another location locally, or a mounted location on a shared file system. You may install PBSA in the default directory by pressing <ENTER>, or you may select a custom directory by entering a directory and pressing <ENTER>. The default installation directory is:




6.1.11 Licensing options

Choose licensing option you want to use (see section 3.3.4, “Licensing options”, on page 17 for more information). The default is base (HWU is Hyper Works Units).




Enter the appropriate option and press <ENTER>.



6.1.12 License server

Enter the license server and press <ENTER>.

.





6.1.13 Pre-installation summary

As a last step before installation a pre-installation summary will be displayed. If the listed specifications are incorrect, enter "back" and press <ENTER> to go back and make any nec-essary changes. If all specifications are correct, proceed with the PBSA installation by press-ing <ENTER>.

IMPORTANT - Please note the port number of the PBSA service. This information will be needed for opening the port for incoming TCP traffic.



6.1.14 Installation progress

You can monitor the progression of the installation by viewing the console progress bar.

6.1.15 Start PBSA services

If you chose to install the demonstration database, then during the installation process a mes-sage will be displayed asking if the PBSA services should be started.

If a license server was not configured for PBSA during installation, do no start the PBSA ser-vice. You will need to configure the license server prior to starting the PBSA service. Enter "2” and press <Enter>.

Otherwise, enter "1" and press <Enter> to start the PBSA service.

If the PBSA service is not started, then you must manually start it once the installation process is complete - see section 14.1.1, “Starting the PBSA service”, on page 141.

See section 17.3, “Troubleshooting the PBSA service startup”, on page 231 for any problems experienced during PBSA service start-up.



6.1.16 Confirm PBSA services should not be started

If you choose to defer the startup of the PBSA service, you will be asked to confirm your choice. Enter "1" and press <ENTER> to confirm this action.

6.1.17 Installation complete

Once the installation of PBSA is complete the installation details will be displayed. Press <ENTER> to exit the installation.



6.1.18 Post-installation

6.1.18.1 PBSA is configured to use the demonstration database

If you have configured PBSA to use the demonstration database, and did not provide a license server during installation, then you must now configure PBSA with a valid license server before proceeding (see section 19.3.1, “Initial license server configuration”, on page 306).

Once a valid license server is known to PBSA, the Web Application is available for use. Use any of the supported browsers (see section 1.2, “Supported Platforms and Browsers”, on page 4) to login to PBSA through the URL http://<hostname>:9000/pbsworks or https://<hostname>:9143/pbsworks.


6.1.18.2 PBSA is configured to use the PBS accounting logs

If you have configured PBSA to use the PBS Professional accounting logs, please complete the post-installation activities (see Chapter 8, "Post-installation steps", on page 105) before using the PBSA Web Application.

6.2 Installing on separate servers

This deployment option is required for sites which want to analyze more than one PBS Pro-fessional complex:

• The PBSA Web Application and Parser are installed on a separate machine from the Data Collectors.

• A PBSA Data Collector must be installed either on the head node of each PBS Profes-sional complex, or on a machine that has access to the PBS Professional accounting logs for that complex.





• adherence to the minimum system requirements (see section 1.1, “System Require-



ments”, on page 2).


To install PBSA download the installer appropriate for your platform from the user area.

6.2.2 Installing the PBSA Web Application and Parser







6.2.2.3 License agreement





Select the PBSA Web Application and Parser component by entering “1” and pressing <ENTER>.

Only the PBSA Web Application and Parser will be installed.



6.2.2.5 Confirm PBSA Web Application and Parser install

A message will be displayed stating that you are installing only the PBSA Web Application and Parser. Enter “1” and press <ENTER> to confirm this action.

6.2.2.6 Dataset

Configure PBSA to use your site’s accounting logs. Do not select the demonstration database option. Enter “2”. and press <ENTER>.



6.2.2.7 Authentication server

PBSA uses SSH to verify credentials of users who login. You can either specify the SSH server running on the local machine, or specify a different host running SSH. Press <ENTER> to proceed with the default authentication server, or specify the SSH server host-name, and press <ENTER>.

6.2.2.8 PBSA service user

Enter a user id that will be used to run the PBSA services. This user must be a pre-existing and preferably non-privileged user on the server where PBSA is being installed. Press <ENTER>.






Specify a location where the selected PBSA components will be installed. These components can be installed to the default location, another location locally, or a mounted location on a shared file system. You may install PBSA in the default directory by pressing <ENTER>, or you may select a custom directory by entering a directory and pressing <ENTER>. The default installation directory is:




6.2.2.10 Licensing options

Choose the licensing option you want to use (see section 3.3.4, “Licensing options”, on page 17 for more information). The default is base (HWU is Hyper Works Units).




Enter the appropriate option and press <ENTER>.



6.2.2.11 License server

Enter the license server and press <ENTER>.






As a last step before installation a pre-installation summary will be displayed. If the listed specifications are incorrect, enter "back" and press <ENTER> to go back and make any nec-essary changes. If all specifications are correct, proceed with the PBSA installation by press-ing <ENTER>.

IMPORTANT - Please note the port number of the PBSA service. This information will be needed during the installation of the PBSA Data Collectors, and for opening the PBSA service port for incom-ing TCP traffic




You can monitor the progression of the installation using the console progess bar. .


Once the installation of the PBSA Web Application and Parser is complete, the installation details will be displayed.

Now the installation of the PBSA Data Collectors on each of the PBS Professional complexes must be completed (see section 6.2.3, “Installing the PBSA Data Collector”, on page 91).

Press <ENTER> to exit the installation.



6.2.3 Installing the PBSA Data Collector

These installation steps must be repeated for each PBS Professional complex. A PBSA Data Collector must be installed either on the head node of each PBS Professional complex, or on a machine that has access to the PBS Professional accounting logs for that complex.









6.2.3.3 License agreement



Select the PBSA Data Collector component by entering “2” and pressing <ENTER>.

Only the PBSA Data Collector will be installed.



6.2.3.5 Confirm PBSA Data Collector install

A message will be displayed stating that you are installing only the PBSA Data Collector. Enter "1" and press <ENTER> to confirm this action.

6.2.3.6 PBSA Web Application and Parser machine location

Specify the hostname of the machine where the PBSA Web Application and Parser were installed. Press <ENTER>.



6.2.3.7 PBSA port

Specify the port number of the PBSA service. Press <ENTER>.

6.2.3.8 Accounting logs location

Specify the location of the PBS Professional accounting logs. The accounting logs can be available locally or via a mounted path on a shared file system. You may choose the default location of the accounting logs, or you may select another directory by specifying another directory path and pressing <ENTER>.




You may install PBSA in the default directory by pressing <ENTER>, or you may select a custom directory by entering a directory and pressing <ENTER>. The default installation directory is:





As a last step before installation a pre-installation summary will be displayed. If the listed specifications are incorrect, enter "back" and press <ENTER> to go back and make any nec-essary changes. If all specifications are correct, proceed with the PBSA installation by press-ing <ENTER>. .


You can monitor the progression of the installation using the console progress bar.




Once the installation of the PBSA Data Collector is complete the installation details will be displayed. Press <ENTER> to exit the installation.

Repeat this installation process for each PBS Professional complex.

After Data Collectors are installed on every PBS Professional complex, perform the post installation activities (see Chapter 8, "Post-installation steps", on page 105) to configure PBSA and make it ready for use.


Chapter 7

Uninstalling PBSAThe uninstallation process must be repeated on each machine where a PBSA component was installed.

If all three components (Web Application, Parser, and Data Collector) have been installed on a single machine, all three components will be uninstalled. You cannot uninstall components separately.

7.1 Uninstalling using the GUI mode

7.1.1 Starting the uninstaller

Navigate to the PBSA installation directory. The installation directory for a typical installa-tion of PBSA is:


Enter the following at the Linux command line prompt (command must contain spaces with escape characters):

cd _PBS\ Analytics_installation

Prior to uninstalling, stop the PBSA service and Data Collector ser-vices.


Chapter 7 Uninstalling PBSA

Execute the uninstall script by entering the following command at the Linux command line prompt (command must contain spaces with escape characters)

./Change\ PBS\ Analytics\ Installation -i swing

7.1.2 Splash screen

After extracting the files, the uninstaller will proceed by displaying the splash screen..


Uninstalling PBSA Chapter 7

7.1.3 Starting the uninstallation process

Click Next to uninstall PBSA.



7.1.4 Uninstall PBSA

Please read the instructions before proceeding with the uninstall. Click Uninstall to initiate the uninstall process. To cancel the uninstall process, click Cancel.

7.1.5 Uninstallation progress

You can monitor the progess of the uninstall process via the uninstall progress bar.


Uninstalling PBSA Chapter 7

7.1.6 Completing the uninstallation process

Items that could not be removed are listed. Please delete them manually. To complete the uninstallation, click Done.

7.2 Uninstalling using the console mode

7.2.1 Starting the uninstaller

Navigate to the PBSA installation directory. The installation directory for a typical installa-tion of PBSA is:


Enter the following at the Linux command line prompt (command must contain spaces with escape characters):

cd _PBS\ Analytics_installation

Execute the uninstall script by entering the following command at the Linux command line prompt (command must contain spaces with escape characters)

./Change\ PBS\ Analytics\ Installation -i console



7.2.2 Starting the uninstallation process

The console uninstaller will launch. Press <ENTER> to continue with the uninstall.

7.2.3 Completing the uninstallation process

Items that could not be removed are listed. Please delete them manually.


Chapter 8

Post-installation stepsAfter installation of PBSA is done, complete the following post-installation activities:

1. Set up the system path

2. Open the PBSA service port

3. Reset the database user password

4. Provide a license server if one was not supplied during installation

5. Configure PBSA

6. Specify shared privileges

7. Start PBSA

8. Parse the accounting logs

9. Correct recovery records

10. Analyze unsupported records

11. Login to the PBSA Web Application


Chapter 8 Post-installation steps

8.1 Setting up the system path

A number of commands are provided for configuring PBSA. Update the system path with the absolute path of these commands.

8.1.1 Single machine deployment

If the Web Application, Parser, and Data Collector have been deployed on a single machine, then execute the following command:

export PATH=$PATH:INSTALL_DIR_13.0/portal/scripts/pbsa-commands/bin:INSTALL_DIR_13.0/dc/scripts/pbsa-commands/bin

8.1.2 Multiple machine deployment

If the Web Application and Parser have been deployed on a separate machine than the Data Collectors, then execute the following command on the machine where the Web Application and Parser have been deployed:

export PATH=$PATH:INSTALL_DIR_13.0/portal/scripts/pbsa-commands/bin

Execute the following command on all machines where the Data Collector has been deployed:

export PATH=$PATH:INSTALL_DIR_13.0/dc/scripts/pbsa-commands/bin


Post-installation steps Chapter 8

8.2 Opening the PBSA service port

Ensure the port assigned to PBSA is open for incoming TCP traffic. The port information is displayed in the “Pre-Installation Summary” screen and “Installation Complete” screen during installation. This port must be opened for incoming TCP traffic to allow communica-tion between the PBSA service and the Data Collector services. If the port is not opened, then the following error will be seen in the Data Collector log file when the Data Collector attempts to transport the accounting log information to the machine where the PBSA Web Application and Parser are installed.:

8.3 Resetting the database user password

Changing the PBSA database password from the default of “pbsworks@123” to a secure site-specific password is highly recommended. The database password can be changed using the pbsa-data-password command - see section 15.9, “Changing the PBSA database pass-word”, on page 206. The PBSA service must be stopped before issuing the pbsa-data-password command. If this command is executed when the PBSA Web Application and database are running, then this command will exit with an error - "ERROR: Please make sure PBSA engine and Monetdb Server are not running."

ERROR, 11 Sep 13 01:48:25,332 -{DataTransportManager.java: 91} - Not able to send file 20110503

com.altair.pbsworks.dc.framework.DCException: File Transport proxy encountered a problem while processign request.

at com.altair.pbsworks.dc.transport.http.HTTPFileTransport.sendData(HTTPFileT-ransport.java:113)

at com.altair.pbsworks.dc.transport.DataTransportManager.transportFile(DataTrans-portManager.java:83)

at com.altair.pbsworks.dc.transport.DataTransportManager.doWork(DataTransportMan-ager.java:73)

at com.altair.pbsworks.dc.framework.AbstractWorker.run(AbstractWorker.java:79)

at java.lang.Thread.run(Thread.java:662)

Caused by: java.io.IOException: Server returned HTTP response code: 500 for URL: http://192.168.33.244:9000/parser/upload

at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnec-tion.java:1436)


... 4 more



8.4 Providing a license server

If you did not supply a license server during installation, then you must now specify a license server before proceeding. See section 19.3.1, “Initial license server configuration”, on page 306 for instructions on how to configure the license server.

8.5 Configuring PBSA

With PBSA 13.0, supplying the following information is now optional.


• holidays


• exit codes

• custom groups


When PBS Professional node information is not supplied, the data displayed in the PBSA charts is confined to the cluster’s job activity (extracted from the PBS Professional accounting logs). You may supply node information at any time and after the next Parser cycle completes, PBSA provides actual values for the above metrics, producing richer, more meaningful charts. For a more complete explanation see Optional Node Configuration Improves Installation Pro-cess on page xi.

PBSA provides defaults for holidays, prime/non-prime time hours, and custom groups for the last five years. It also supplies defaults for exit codes. These defaults are summarized in Table 10-1, “Configuration defaults”. You may use these defaults or provide site specific information.

Similar to the node configuration, PBSA 13.0 now allows holiday, prime time, and non-prime time information to be changed at any time and once the next Parser cycle completes, the PBSA charts will reflect those changes without having to reparse the accounting logs. For more information see Reparsing Not Required for Holiday, Prime, and Non-Prime Time Changes on page xiv.

Follow these steps to configure PBSA with this optional information:

1. Supplying PBS node information on page 113

2. Supplying other site details on page 119



8.6 Specify shared privileges

Specify the list of users who have privileges to add, delete, and modify charts and dashboards in the “Shared” category.

1. Edit the file:INSTALL_DIR_13.0/portal/services/pbsa/config/pbsacon-fig.properties

2. Specify the list of users who have privileges to add, delete, and modify charts and dashboards in the “Shared” category by adding comma separated user names sur-rounded by double-quotes to the following line:

By default, the PBSA service user has privileges to add, delete, and modify charts and dashboards in the “Shared” category.

3. If the PBSA service is running, then restart the service.

8.7 Starting PBSA

Once the installation and optional configuration of PBSA is complete, PBSA can be started. The PBSA service should be started first. Once the PBSA service has been started, the Data Collectors can be started. For more information about starting PBSA see section , “Starting PBSA”, on page 127.

Be aware, if there are any PBS Professional accounting logs awaiting parsing, then these log files will be parsed after the PBSA and Data Collector services are started or restarted.

reportSharingOwners=root,esmith,jtaylor



8.8 Parsing the accounting logs

Once the PBSA service is started, and the delay for processing the accounting log has expired, the Parser will begin processing any PBS Professional accounting log information that has been made available by the Data Collector.

The Web Application will not be available until parsing is complete. If you try to access the Web Application before parsing is complete, a Progress Indicator Screen is displayed contain-ing the following message:

Accounting log parsing is not yet complete. Please log out. Login once parsing is com-plete. View the Parser log file to track the parsing progress - see the PBSA Systems Administrator’s Guide section “Parsing the accounting logs”.

The Progress Indicator Screen allows a user to visualize the progression of the parsing and caching process.

For more information about parsing the accounting logs see Chapter 12, "Parsing the account-ing logs", on page 131.

8.9 Correcting recovery records

There are some situations where PBSA is unable to process a PBS Professional accounting log record due to an issue with the value of a PBS attribute. When the Parser finds an issue with an accounting log entry, the entry is written to file in a recovery directory. A recovery directory is created for each Data Collector. The recovery directories are located in:

INSTALL_DIR_13.0/portal/services/pbsa/parser/modules/pbsdata

For example, an accounting log called 20150622 is transported from a Data Collector hav-ing a FQDN of server.prog.nt.com. PBSA finds an issue with a record in the account-ing log and creates a file having the same name in the following directory:

INSTALL_DIR_13.0/portal/services/pbsa/parser/modules/pbsdata/server_prog_nt_com/recovery



PBSA will write the accounting log record to this file and log a message to the parser.log file, for example:

The PBSA service user can edit this file and correct the issue with the accounting log entry. The corrected entries are parsed and loaded into the PBSA database during the next Parser cycle.

ERROR, 11 Jun 2015 21:29:12,311 -{ParsedRescourceConatinerEventResourceOccupan-cyDetails.java: 486} - Error while parsing for unknown reasons : 12/20/2012 23:35:37;S;81960.blrhpc3;user=manjub group=engr jobname=UF_newB_FC_11 queue=chrysler-abaqus ctime=1356010027 qtime=1356010027 etime=1356010027 start=1356026737 exec_host=blrh-pc3n9/0*4 exec_vnode=blrhpc3n9:abaqus=8:mem=30720000kb:ncpus=4 Resource_List.abaqus=8 Resource_List.ecid=51097 Resource_List.mem=30000mib Resource_List.ncpus=4 Resource_List.nodect=1 Resource_List.place=pack Resource_List.platform=Linux Resource_List.select=1:abaqus=8:mem=30000mb:ncpus=4:platform=Linux Resource_List.soft-ware=abaqus resource_assigned.mem=30720000kb resource_assigned.ncpus=4

ERROR, 11 Jun 2015 21:29:12,312 -{ParsedRescourceConatinerEventResourceOccupan-cyDetails.java: 487} - Exception Trace :

java.lang.NullPointerException

at com.altair.pbsworks.parser.transform.ParsedRescourceConatinerEventResourceOccupancyDetails.convertValues(ParsedRescourceConatinerEventResourceOccupancyDetails.java:301)

at com.altair.pbsworks.parser.transform.RescourceConatinerEventResourceOccupancyDetailsPopulatorThread.run(RescourceConatinerEventResourceOccupancyDetailsPopulatorThread.java:48)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)


INFO, 11 Jun 2015 21:29:12,317 -{ParsedRescourceConatinerEventResourceOccupan-cyDetails.java: 488} - Record will be written into recovery

WARN, 11 Jun 2015 21:29:12,403 -{RescourceConatinerEventResourceOccupancyDetailsPopulator-Thread.java: 69} - Record will be written into recovery : 12/20/2012 23:35:37;S;81960.blrh-pc3;user=manjub group=engr jobname=UF_newB_FC_11 queue=chrysler-abaqus ctime=1356010027 qtime=1356010027 etime=1356010027 start=1356026737 exec_host=blrhpc3n9/0*4 exec_vnode=blrh-pc3n9:abaqus=8:mem=30720000kb:ncpus=4 Resource_List.abaqus=8 Resource_List.ecid=51097 Resource_List.mem=30000mib Resource_List.ncpus=4 Resource_List.nodect=1 Resource_List.place=pack Resource_List.platform=Linux Resource_List.select=1:abaqus=8:mem=30000mb:ncpus=4:platform=Linux Resource_List.soft-ware=abaqus resource_assigned.mem=30720000kb resource_assigned.ncpus=4



8.10 Analyzing unsupported records

There are some situations where an accounting log record fails the validation criteria of PBSA:

• Binary data present in the accounting log record

• Main job information of a job array

• Data not available for user, group, start time, queued time, end time, memory used, and execution host.

When an accounting log record fails validation, the entry is written to file in a unsup-ported directory. A unsupported directory is created for each Data Collector. The unsupported directories are located in:

INSTALL_DIR_13.0/portal/services/pbsa/parser/modules/pbsdata

View the contents of a file in the unsupported directory to analyze the issue. You may need to make changes to the PBS Professional configuration to make the missing/invalid PBS attribute available in the accounting log record.

8.11 Logging into the PBSA Web Application

Caching and initialization of the PBSA engine is complete and the Web Application is avail-able when you see this message in the pbsworks.log file:

You may now login to the PBSA Web Application by using any of the supported web brows-ers through the following URL’s:

• http://<hostname>:9000/pbsworks

• https://<hostname>:9143/pbsworks


See section 17.4, “Troubleshooting logging into PBSA”, on page 233 to solve problems expe-rienced while logging into PBSA.

INFO, 08 Jun 2015 13:52:24,474 -{CachePrimingWorker.java: 114} - Initialization of the ana-lytics engine completed successfully in 0 minutes. You may now login to the PBSA web appli-cation using the URL http://<hostname>:<port>/pbsworks


Chapter 9

Supplying PBS node informationProviding information about the nodes of your PBS cluster is optional, however it improves the reporting capabilities of PBSA. PBSA does this by storing a detailed node configuration map for each PBS complex.

When PBS Professional node information is not supplied, the data displayed in the PBSA charts is confined to the cluster’s job activity (extracted from the PBS Professional accounting logs). You may supply node information at any time and after the next Parser cycle completes, PBSA provides actual values for the above metrics, producing richer, more meaningful charts without having to reparse the accounting logs. For a more complete explanation see “Optional Node Configuration Improves Installation Process” on page xi.

Follow these steps to provide node information for each PBS Complex:

1. obtain PBS complex node information using the pbsnodes -av command or use the pbsa-setup-counsellor command to create the pbsnodes -av output file. For more information about the pbsa-setup-counsellor command see section 15.1, “Analyzing a PBS Professional complex”, on page 149.

2. evaluate nodes that are offline or down using the pbsa-setup-counsellor command to provide information about offline or down nodes.

3. use the pbsa-setup-counsellor command to provide information about nodes that are found in the accounting log, but are missing from the pbsnodes -av output. These nodes should be manually added to the pbsnodes -av output file, otherwise PBSA cannot account for those accounting log entries specific to those nodes. PBSA will write these log entries to a file in a recovery directory.

4. run the PBSA node configuration command pbsa-config-nodes

5. repeat steps 4 for every year for which the PBS Complex has accounting log data


Chapter 9 Supplying PBS node information

9.1 Obtain node information

To build a node configuration map for a PBS cluster, obtain node information by executing the PBS Professional command pbsnodes -av for each PBS Professional complex. The output of this command should be written to an output file, for instance:

pbsnodes -av > pbsnodes1.out

Or you can use the pbsa-setup-counsellor command to create the pbsnodes -av output file.

The data in the pbsnodes -av output file is basically a snapshot of the current PBS com-plex node configuration. PBSA assumes that your site’s nodes have always been configured this way, and will provide utilization statistics based on those details.

The data contained in these output files is used as input to a PBSA configuration command, pbsa-config-nodes.


Supplying PBS node information Chapter 9

9.2 Evaluate offline or downed nodes

You can use the pbsa-setup-counsellor command to provide information about offline or down nodes. In addition, the pbsa-config-nodes command will return a mes-sage similar to the following if any nodes are offline or down:

This message will be returned for nodes having a comment equal to “node down:communica-tion closed”, or having the following states:

• down,unresolvable

• state-unknown,offline

• state-unknown,down

The system administrator must perform some analysis at this point to determine the cause of the downed node. For instance, there may be a communication problem between the PBS Professional MOM and headnode that is causing the node to be marked as down, when actu-ally the node is up and running. In this case correct the communication problem, verify that node resources are correct, re-run the pbsnodes -av command, and issue the pbsa-config-nodes command again.

If the node is down for a valid reason, such as maintenance, then you will have to manually edit the pbsnodes -av output file. You may:

• change the state of the node to an online state such as “free”. If you choose to set the state of a downed node to “free”, verify that the following two parameters are present in the pbsnodes -av output for that node:

• resources_available.ncpus

• resources_available.mem

If these parameters are not present, then manually enter this information for that node.

• delete the node from the pbsnodes -av output file. You can configure PBSA at a later date when the node comes online (see section 12.5, “Configuration changes after initial parsing”, on page 137). However, deleting the node from the file creates a situation where PBSA is “unaware” of this node during parsing of the accounting logs. PBSA cannot account for those accounting log entries specific to that node, therefore PBSA will write these log entries to a file in a recovery directory.

Please check the pbsanodes-av output file, some of the nodes are offline. Following nodes are offline: ['tribm3550m3-02']



9.3 Run the node configuration command

The pbsa-config-nodes command will need to be executed for each PBS Professional complex, that is for each pbsnodes -av output file. The pbsa-config-nodes com-mand must be executed on the machine where the Parser is deployed.

The following information is mandatory input for the pbsa-config-nodes command, and can be obtained by running the pbsa-setup-counsellor command:

• the PBS Professional server hostname as specified in the PBS Professional accounting logs

• the fully qualified domain name of the machine where the Data Collector is installed

• the output from the PBS Professional command: pbsnodes -av.

The basic usage for the pbsa-config-nodes command is:

pbsa-config-nodes PRIMARY_SERVER SECONDARY_SERVER DC_FQDN PATH_TO_PBSNODES.OUT --start-date=STARTDATE --end-date=ENDDATE --furnishing-servers=FUR_SERVER

where:

PRIMARY_SERVER is the name of the PBS Professional primary server as specified in the PBS Professional accounting logs

SECONDARY_SERVER (optional) is the name of the PBS Professional failover server

DC_FQDN is the fully qualified domain name of the machine where the Data Collector is installed.

PATH_TO_PBSNODES.OUT is the output file from the command: pbsnodes -av

--start-date,--end-date is the date range when the node configuration is in effect

--furnishing-servers is a comma separate list of furnishing servers if peer scheduling is implemented. See Chapter 22, "Peer Scheduling", on page 333 for more information.

The PRIMARY_SERVER must be the server name as specified in the PBS accounting logs.


Supplying PBS node information Chapter 9

For example, if a site has three PBS Professional complexes: complex A, complex B, and complex C, then execute the following commands for:

Complex A:

pbsnodes -av > pbsnodes_complexA.out

Complex B:

pbsnodes -av > pbsnodes_complexB.out

Complex C.:

pbsnodes -av > pbsnodes_complexC.out

Execute these commands on the server where the PBSA Parser is installed:

pbsa-config-nodes serverA serverA.ms.com pbsnodes_complexA.out --start-date=2015-01-01

pbsa-config-nodes serverB serverB.ms.com pbsnodes_complexB.out --start-date=2015-01-01

pbsa-config-nodes serverC serverC.ms.com pbsnodes_complexC.out --start-date=2015-01-01

These commands will create a node details file for the year 2015, making it effective from January 1, 2015 until “now” (--end-date defaults to “now” which means the node configura-tion is in effect going forward). For more information about the node details file see section 18.5, “Node details file”, on page 275.



9.4 Node configuration for previous years

You will need to create a node details file for every year for which your site has accounting log data.

For example, if your site has accounting log data for the years 2012-2014 in addition to the year 2015, then you must create a node details file for the years 2012, 2013, and 2014.

Execute the following commands to create a node details file for 2012 for each PBS Profes-sional complex:

pbsa-config-nodes serverA serverA.ms.com pbsnodes_conplexA.out --start-date=2012-01-01 --end-date=2012-12-31

pbsa-config-nodes serverB serverB.ms.com pbsnodes_conplexB.out --start-date=2012-01-01 --end-date=2012-12-31

pbsa-config-nodes serverC serverC.ms.com pbsnodes_complexC.out --start-date=2012-01-01 --end-date=2012-12-31









Executing these commands assumes that the node configuration did not change from 2012 to 2014.


Chapter 10

Supplying other site detailsVarious default charts display job statistics for prime hours, non-prime time hours, custom groups, and job failure rates. Providing the following site specific information can improve the data presented in these default charts:

• holidays


• exit codes

• custom groups


PBSA provides defaults for holidays, prime/non-prime time hours, and custom groups for the last five years. It also supplies defaults for exit codes. These defaults are summarized in the table below. You may use these defaults or provide site specific information.

PBSA 13.0 now allows holiday, prime time, and non-prime time information to be changed at any time and once the next Parser cycle completes, the PBSA charts will reflect those changes without having to reparse the accounting logs. For more information see “Reparsing Not Required for Holiday, Prime, and Non-Prime Time Changes” on page xiv.


Chapter 10 Supplying other site details

The following sections explain how to provide this information. If you prefer to use the defaults, then you can skip these steps and start PBSA.

1. Configuring holidays

2. Configuring prime and non-prime hours

3. Configuring custom groups

4. Configuring exit codes

5. Configuring custom resources

Table 10-1: Configuration defaults

Configuration Year(s) Default

Holiday 2011-2015 New Year’s Day

Prime

Non-prime time

2011-2015 Monday - Friday

Saturday

Sunday

Prime time begins at 09:00 am

Non-prime time begins at 5:30 pm

Day is scheduled as non-prime time


Exit codes N/A Successful

Unsuccessful

0

Refer to section 18.2.3, “Default exit code configuration data”, on page 266.

Custom group 2011-2015 “OTHERS” Required by PBSA - do NOT delete

PBSA defines prime and non-prime time as:

Prime Time : 08:00 AM to 17:30 PM Non Prime Time Morning: 12:00 AM to 8:00 AM Non Prime Time Evening : 17:30 PM to 12:00 AM


Supplying other site details Chapter 10

10.1 Configuring holidays

By default, PBSA is configured with a single holiday, New Year’s day, for the years 2011 through 2015.

Use the command pbsa-config-holidays to add holiday information to PBSA. This command must be executed on the machine where the Parser is deployed.

For instance, the following command will configure a holiday that takes place over a single day.

pbsa-config-holidays MemorialDay --start-date=2015-05-25 --enddate=2015-05-25

The following command shows how to configure a holiday that takes place over multiple days.

pbsa-config-holidays Thanksgiving --start-date=2015-11-26 --end-date=2015-11-27

The following shows how to configure a holiday that takes place in a previous year.


Default entries may be removed using the pbsa-config-holidays --remove option.

Once the accounting logs are parsed for the first time, any changes to the PBSA holiday infor-mation will be incorporated in the next parsing cycle. You may restart the PBSA service to initiate a parsing cycle immediately if necessary.

For more information about using the pbsa-config-holidays command see section 15.3, “Configuring holidays”, on page 164.



10.2 Configuring prime and non-prime hours

By default, PBSA is configured with prime and non-prime time hours for the years 2011 through 2015. Prime times hours for Monday through Friday begin at 09:00 am. Non-prime time hours begin at 5:30 pm. For Saturday and Sunday, the entire day is scheduled as non-prime time.

Use the command pbsa-config-holidays to add prime and non-prime time informa-tion to PBSA. This command must be executed on the machine where the Parser is deployed.

For instance, the following command configures prime time to start at 9:00 am and non-prime to start at 6:00 pm for Monday.

pbsa-config-holidays monday --prime-time=09:00 --non-prime-time=18:00

Executing the following command configures prime time to start at 8:00 am and non-prime time to start at 5:30 pm for Monday. Prime time start defaults to 8:00 am and non-prime start defaults to 5:30 pm (17:30).

pbsa-config-holidays monday

Executing this command configures prime and non-prime time hours for any weekdays (Mon-day-Friday) not specified separately. For instance, if prime time and non-prime time have already been specified for the days Monday and Tuesday, then this command will specify prime time and non-prime time for days Wednesday through Friday.

pbsa-config-holidays weekdays --prime-time=09:00 --non-prime-time=18:00

Executing the following command configures prime time to start at 9:00 am and non-prime time to start at 6:00 pm for weekends (Saturday and Sunday).

pbsa-config-holidays weekends --prime-time=09:00 --non-prime-time=18:00

This following command configures prime time to start at 9:00 am and non-prime to start at 6:00 pm for Monday for the year 2014 (a previous year’s configuration).

pbsa-config-holidays monday --prime-time=09:00

--non-prime-time=18:00 --start-date=2014-01-01

Default entries may be removed using the pbsa-config-holidays --remove option.

Once the accounting logs are parsed for the first time, any changes to the PBSA prime/non-prime time information will be incorporated in the next parsing cycle. You may restart the PBSA service to initiate a parsing cycle immediately if necessary.

For more information about using the pbsa-config-holidays command see section 15.3, “Configuring holidays”, on page 164.



10.3 Configuring custom groups

Custom groups can be used to track utilization at a project level for sites running versions of PBS Professional prior to 11.2. The preferred method for tracking utilization at a project level is the use of the -P option when submitting a job via the qub command. For more infor-mation see section 3.2.3, “Utilizing the qsub -P option”, on page 15.

Use the command pbsa-config-groups to add this information to PBSA. This com-mand must be executed on the machine where the Parser is deployed.

For example, several engineers may be assigned to a project. To track the total number of jobs executed for that project, PBSA must be configured so that it knows which users are assigned to that project.

The following command configures a user “tsmith” to be assigned to a custom group (i.e., project) “CFD” from January 22, 2015 to November 23, 2015.

pbsa-config-groups tsmith CFD --start-date=2015-01-22 --end-date=2015-11-23

There are some caveats for entering this information, so it is recommended to review section 15.4, “Configuring custom groups”, on page 175 for more information about using the pbsa-config-groups command.

PBSA is configured with a default group “OTHERS” for the years 2011 through 2015. Do not delete this default group.

Once the accounting logs are parsed for the first time, modifications to the current year's information should only be done in two situations and do not require any reparsing of accounting logs.

1. A user is assigned to a new project.Use the pbsa-config-groups command to assign the user to the new project. Set the start date to today's date or later. The end date can be set to a future date or "now".

2. A user completes an assignment.

Use the pbsa-config-groups command to update the user’s completion date by setting the end date to the assignment completion date.

These custom group changes will be incorporated in the next parsing cycle. You may restart the PBSA service to initiate a parsing cycle immediately if necessary.

If the custom group information is modified for any other reason, then any parsed data must be discarded and all PBS Professional accounting logs have to be reparsed.



10.4 Configuring exit codes

In order to track job execution failure rates and reasons for failures, it is recommended that PBSA be configured with the exit codes that are used at your site. PBSA is configured by default with two exit reasons “Successful” and “Unsuccessful” that are mapped to specific exit codes. An example of this default mapping can be found in section 18.2.3, “Default exit code configuration data”, on page 266.

Use the command pbsa-config-exits to add exit code information to PBSA. This com-mand must be executed on the machine where the Parser is deployed.

For instance, the following command configures an exit code of “1” to be defined as “Suc-cessful” for the application PBSJobApp.

pbsa-config-exits 1 --reason=Successful --application=PBSJobApp

Executing the following command configures a list of exit codes to be defined as “Unspeci-fied” (default) for all applications (default).

pbsa-config-exits 2,143,255

PBS exit codes can be found in the “Job Exit Codes” section of the PBS Professional Admin-istrator’s Guide.

If changes are made to exit codes after the initial parsing cycle is complete, any parsed data must be discarded and all PBS Professional accounting logs have to be reparsed.

For more information about using the pbsa-config-exits command see section 15.5, “Configuring exit codes”, on page 183.

The default exit code “Still-Running” must not be removed.



10.5 Configuring custom resources

PBSA reports on various common PBS Professional resources such as CPUs, memory, and walltime. In addition, PBSA can report upon site-specific resources known as custom resources. This type of reporting requires a PBSA Hook, a mechanism for adding a custom resource to PBSA. Once a custom resource is configured, it can be added to a chart either as a Dimension or a Measure. For more information about creating a PBSA Hook see Chapter 23, "PBSA Hooks", on page 335.

If changes are made to PBSA Hooks after the initial parsing cycle is complete, any parsed data must be discarded and all PBS Professional accounting logs have to be reparsed.


Chapter 11

Starting PBSA Follow these steps to start PBSA after installation and configuration are complete:

1. Starting the PBSA service

2. Starting the Data Collectors

11.1 Starting the PBSA service

Login to the server where the PBSA Web Application and Parser has been deployed using the login credentials of the PBSA service user specified during the installation of PBSA. Start the PBSA service using the command:

/etc/init.d/pbsworks-pbsa start

You may check the status of the PBSA service using the PBSA status command:

/etc/init.d/pbsworks-pbsa status

For more information about the PBSA service commands see Chapter 14, "PBSA service commands", on page 141.

For any problems experienced while starting the PBSA service see section 17.3, “Trouble-shooting the PBSA service startup”, on page 231.


Chapter 11 Starting PBSA

11.2 Starting the Data Collectors

The PBSA Data Collector makes the PBS Professional accounting logs accessible to the PBSA Parser. Every minute the Data Collector performs the following tasks:

• it copies the accounting logs from your site’s PBS Professional accounting log location to an intermediate holding area

• it transports the information stored in the holding area to the machine where the PBSA Web Application and Parser are installed

Depending on the deployment option selected, you may have a single Data Collector, or you may have multiple Data Collectors installed across separate servers.

11.2.1 Deployment Option 1 - single server

If you have deployed the Data Collector, Parser, and the Web Application on a single server, then only a single Data Collector has been installed.

Login to the server where PBSA has been deployed using the login credentials of the PBSA service user specified during the installation of PBSA. Start the PBSA Data Collector using the command:

/etc/init.d/pbsworks-dc start


11.2.2 Deployment Option 2 - separate servers

If you have deployed the Web Application and Parser on a separate server from the Data Col-lector, then multiple Data Collectors may have been installed.

Login to each server where a Data Collector has been deployed using the login credentials of the PBSA service user specified during the installation of PBSA. Start the PBSA Data Col-lector using the command:


Continue this process until all Data Collectors have been started.

With PBSA 13.0, the Data Collector transport delay time has been reduced to 1 minute.


Starting PBSA Chapter 11


11.2.3 Determining the status of a Data Collector

The Data Collector will log messages to the file INSTALL_DIR_13.0/dc/logs/dc.log. This file is located on the machine where the Data Collector has been installed. Use the following command to view the contents of this file to determine if the Data Collector started successfully.

cat dc.log

If the Data Collector starts successfully, you will see messages similar to this:

Depending on the level of logging, you may also see messages indicating that the Data Col-lector has successfully uploaded PBS Professional log files to the machine where the PBSA Web Application and Parser are installed:

INFO, 08 Jun 2015 13:49:08,943 -{DataCollectorController.java: 62} - Data Collector start-ing.

INFO, 08 Jun 2015 13:49:09,623 -{SimpleThreadPool.java: 253} - Job execution threads will use class loader of thread: main

INFO, 08 Jun 2015 13:49:09,656 -{RAMJobStore.java: 143} - RAMJobStore initialized.

INFO, 08 Jun 2015 13:49:09,659 -{StdSchedulerFactory.java: 892} - Quartz scheduler 'DefaultQuartzScheduler' initialized from default resource file in Quartz package: 'quartz.properties'

INFO, 08 Jun 2015 13:49:09,659 -{StdSchedulerFactory.java: 896} - Quartz scheduler version: 1.4.3

INFO, 08 Jun 2015 13:49:09,660 -{QuartzScheduler.java: 370} - Scheduler DefaultQuartzSched-uler_$_NON_CLUSTERED started.

INFO, 08 Jun 2015 13:49:09,687 -{DataCollectorController.java: 88} - Data Collector started successfully.

DEBUG, 08 Jun 2015 13:50:09,790 -{HTTPFileTransport.java: 123} - Successfully uploaded file 20150201





Chapter 11 Starting PBSA


Chapter 12

Parsing the accounting logs

The parsing of the accounting logs is necessary to make PBS Professional job data available to the PBSA Web Application. The Parser reads the PBS Professional accounting logs, parses the appropriate information from the accounting logs, and then loads this information into a database used by the Web Application.

Once the PBSA service is started, and the delay for processing the accounting log has expired, the Parser will begin processing any PBS Professional accounting log information that has been made available by the Data Collector.

Events and errors that occur while parsing the node detail and holiday configuration files are written to the PBS Works log file (pbsworks.log). All other events and errors that occur while parsing are written to the Parser log file (parser.log). These two log files are located on the machine where the Parser is installed in the following directory:

INSTALL_DIR_13.0/portal/logs

12.1 Tracking the parsing progress

Use the Linux cat command to view the contents of the parser.log log file to track the progress of the parsing process:

With PBSA 13.0, the Parser delay time has been reduced from 30 minutes to 15 minutes.


Chapter 12 Parsing the accounting logs

If the Parser starts successfully, you will see the following message:

If you did not configure any custom resources (PBSA Hooks), messages similar to the follow-ing are displayed:

If you did configure custom resources (PBSA Hooks), messages similar to the following are displayed:

If you did not provide node detail information, messages similar to the following are dis-played:

INFO, 08 Jun 2015 13:48:56,672 -{AnalyticsApplication.java: 467} - **************************Starting PBS Analytics Parser component**************************

INFO, 08 Jun 2015 13:48:57,635 -{CustomResourceProcessor.java: 49} - Going to read custom resource configuration

INFO, 08 Jun 2015 13:48:57,696 -{CustomResourceDisplaySchema.java: 323} - ************************* No custom resource configured *************************

INFO, 08 Jun 2015 13:48:57,704 -{CustomResourceProcessor.java: 147} - Custom Resource pro-cessing finished

INFO, 08 Jun 2015 15:29:46,950 -{CustomResourceProcessor.java: 49} - Going to read custom resource configuration

INFO, 08 Jun 2015 15:29:46,986 -{CustomResourceProcessor.java: 64} - Configuring Resource_List.ngpus custom resource

INFO, 08 Jun 2015 15:29:50,493 -{CustomResourcesDBUtil.java: 108} - Adding custom resource column : ALTER TABLE rc_event_resource_occupancy_base ADD COLUMN resource_list_ngpus bigint

INFO, 08 Jun 2015 15:29:50,608 -{CustomResourceDisplaySchema.java: 318} - Updating custom resources in schema file

INFO, 08 Jun 2015 15:29:50,715 -{CustomResourceProcessor.java: 147} - Custom Resource pro-cessing finished

WARN, 08 Jun 2015 13:48:57,737 -{ConfigurationValidator.java: 27} - nodedetails config file not present

WARN, 08 Jun 2015 13:48:57,737 -{ConfigurationValidator.java: 66} - Not providing these node details configurations will result in product showing you limited set of reports.Please consult Adminstrator Guide for details of charts and features provided / affected by this configuration.


Parsing the accounting logs Chapter 12

Messages similar to the following will be displayed indicating that the Parser has read the cus-tom group and exit code configuration information:

Once the Parser begins parsing the PBS Professional accounting logs you will see messages similar to these:

Followed by these messages indicating that parsing is complete:

INFO, 08 Jun 2015 13:51:57,742 -{CustomGroupCachePopulator.java: 87} - Reading config file : /opt/altair/pbsa/13.0/portal/services/pbsa/parser/modules/pbsdata/config/cus-tom_group_2011.conf

INFO, 08 Jun 2015 13:51:57,746 -{CustomGroupCachePopulator.java: 157} - Finished Reading file : /opt/altair/pbsa/13.0/portal/services/pbsa/parser/modules/pbsdata/config/cus-tom_group_2011.conf

.....

INFO, 08 Jun 2015 13:51:57,759 -{CustomGroupCachePopulator.java: 87} - Reading config file : /opt/altair/pbsa/13.0/portal/services/pbsa/parser/modules/pbsdata/config/cus-tom_group_2016.conf

INFO, 08 Jun 2015 13:51:57,760 -{CustomGroupCachePopulator.java: 157} - Finished Reading file : /opt/altair/pbsa/13.0/portal/services/pbsa/parser/modules/pbsdata/config/cus-tom_group_2016.conf

INFO, 08 Jun 2015 13:51:57,760 -{ExitReasonCachePopulator.java: 70} - Reading config file : /opt/altair/pbsa/13.0/portal/services/pbsa/parser/modules/pbsdata/config/exit_status.conf

INFO, 08 Jun 2015 13:51:57,765 -{ExitReasonCachePopulator.java: 95} - Finished reading file : /opt/altair/pbsa/13.0/portal/services/pbsa/parser/modules/pbsdata/config/exit_sta-tus.conf

INFO, 08 Jun 2015 13:51:59,017 -{Parser.java: 433} - Parsing 20150201 log file

INFO, 08 Jun 2015 13:51:59,055 -{Parser.java: 618} - File is parsed: 20150201

INFO, 08 Jun 2015 13:51:59,074 -{Parser.java: 714} - File 20150201 deleted.

INFO, 08 Jun 2015 14:53:44,996 -{AnalyticsApplication.java: 300} - Parser has completed parsing the accounting logs.

INFO, 08 Jun 2015 14:53:44,996 -{AnalyticsApplication.java: 301} - Analytics engine is now being initialized with the parsed accounting log data. Please do not login to the web application until this process is complete.

INFO, 08 Jun 2015 14:53:44,996 -{AnalyticsApplication.java: 302} - To check for completion of the initialization process, view the web application log file pbsworks.log.



12.2 Tracking the caching and initialization progress

Use the Linux cat command to view the contents of the pbsworks.log log file to track the parsing of the holiday and node detail configuration information, and the progress of the cach-ing and initialization process:

Messages similar to the following will be displayed indicating that the Parser has read the hol-iday configuration information:

These messages indicate PBSA is processing the node details configuration information:

If you did not provide node detail information, messages similar to the following are dis-played:

INFO, 08 Jun 2015 13:49:04,514 -{HolidayConfigCache.java: 62} - Reading config file : /opt/altair/pbsa/13.0/portal/services/pbsa/parser/modules/pbsdata/config/holiday_2011.conf

INFO, 08 Jun 2015 13:49:04,529 -{HolidayConfigCache.java: 90} - Finished reading file : /opt/altair/pbsa/13.0/portal/services/pbsa/parser/modules/pbsdata/config/holiday_2011.conf

.....

INFO, 08 Jun 2015 13:49:04,547 -{HolidayConfigCache.java: 62} - Reading config file : /opt/altair/pbsa/13.0/portal/services/pbsa/parser/modules/pbsdata/config/holiday_2016.conf

INFO, 08 Jun 2015 13:49:04,549 -{HolidayConfigCache.java: 90} - Finished reading file : /opt/altair/pbsa/13.0/portal/services/pbsa/parser/modules/pbsdata/config/holiday_2016.conf

INFO, 08 Jun 2015 13:49:04,553 -{NodeDetailConfigCache.java: 29} - Reading node details file to populate availabilty information

INFO, 08 Jun 2015 13:49:06,686 -{ConfigurationHelper.java: 127} - resource_availability table dropped.

INFO, 08 Jun 2015 13:49:06,700 -{ConfigurationHelper.java: 185} - resource_availability table created.

ERROR, 08 Jun 2015 13:49:06,701 -{ConfigReader.java: 51} - nodedetails file not present, not providing these node details configurations wil result in product showing you limited set of reports.Please consult Adminstrator Guide for details of charts and features pro-vided / affected by this configuration.

ERROR, 08 Jun 2015 13:49:06,716 -{ConfigurationProvider.java: 51} - Could not populate resource availabilty information



If you did provide node detail information, these messages indicate that PBSA is reading and parsing the information:

The following messages indicate that the queries associated with the default PBSA charts are being cached:

Once caching is complete, you will see messages similar to these:

Caching and initialization of the PBSA engine is complete and the Web Application is avail-able when you see this message:

INFO, 08 Jun 2015 14:50:35,415 -{ConfigReader.java: 73} - Reading Config File:/opt/altair/pbsa/13.0/portal/services/pbsa/parser/modules/pbsdata/config/nodedetails_2015.conf

INFO, 08 Jun 2015 14:50:35,425 -{ConfigReader.java: 134} - Finished Reading File:/opt/altair/pbsa/13.0/portal/services/pbsa/parser/modules/pbsdata/config/nodedetails_2015.conf

INFO, 08 Jun 2015 14:50:35,425 -{ConfigurationProvider.java: 47} - Parsing NODE DETAILS Configuration

INFO, 08 Jun 2015 13:52:19,576 -{CachePrimingWorker.java: 105} - OTB Cube caching started...

INFO, 08 Jun 2015 13:52:19,576 -{DefaultCubeCacheStrategy.java: 54} - Discover call started

INFO, 08 Jun 2015 13:52:19,780 -{DefaultCubeCacheStrategy.java: 56} - Discover call com-pleted

INFO, 08 Jun 2015 13:52:19,783 -{DefaultCubeCacheStrategy.java: 58} - OTB Cube cache started

INFO, 08 Jun 2015 13:52:19,864 -{DefaultCubeCacheStrategy.java: 148} - Executing: Success-ful_Vs_Unsuccessful_Jobs_by_Node

INFO, 08 Jun 2015 13:52:19,966 -{CubeBOImpl.java: 109} - Time taken to execute Query is 102 ms

INFO, 08 Jun 2015 13:52:24,376 -{DefaultCubeCacheStrategy.java: 60} - OTB Cube cache com-pleted

INFO, 08 Jun 2015 13:52:24,376 -{CachePrimingWorker.java: 109} - OTB Cube caching completed in 4800 ms

INFO, 08 Jun 2015 13:52:24,474 -{CachePrimingWorker.java: 114} - Initialization of the ana-lytics engine completed successfully in 0 minutes. You may now login to the PBSA web appli-cation using the URL http://<hostname>:<port>/pbsworks



12.3 Initial parsing of the accounting logs

The initial parsing cycle takes place after the installation and configuration of PBSA is com-pleted, and the PBSA and Data Collector services are started for the first time. This initial parsing process may take several hours, depending upon the amount of data stored in the PBS Professional accounting log(s), number of processors, and amount of available RAM.

Once the initial parsing cycle is complete, changes to some PBSA configuration files will require the parsed data to be discarded, and all accounting log data to be reparsed. For more information see section 12.5, “Configuration changes after initial parsing”, on page 137.

For a site that is installing PBSA for the first time, configuration of PBSA followed by pars-ing, may be an iterative process requiring several parsing cycles before PBSA is configured appropriately by the system administrator. If after running a parsing cycle, you find that addi-tional configuration is required, you may discard the data from the previous parsing cycle, modify your configuration, and run the parsing cycle again. For more information see Chap-ter 20, "Reparsing the accounting logs", on page 323.

12.4 Daily parsing of the accounting logs

After the initial parsing of the PBS Professional accounting logs, newly generated accounting logs will continue to be parsed on a daily basis. PBSA is configured by default to run the parsing cycle at 1:00 am, but can be configured to run at a different time. For more informa-tion about adjusting the timing of the parsing cycle see section 15.7, “Configuring the Parser”, on page 195. Currently, this cycle should not be run more than once in a 24-hour period.



12.5 Configuration changes after initial parsing

The following sections describe the effects of making changes to the PBSA configuration files.

12.5.1 Changes that do not require reparsing

Any changes made to the node detail, holiday, and prime/non-prime time information do not require the PBS Professional accounting logs to be reparsed. Instead, your changes will be incorporated in the next parsing cycle. If you do not want to wait until the next parsing cycle to include your changes, you may restart the PBSA service to initiate a parsing cycle.

Modifications to the current year's custom group information that do not require reparsing of the accounting logs are:

1. Assigning a user to a new projectUse the pbsa-config-groups command to assign a user to a new project. Set the start date to today's date or later. The end date can be set to a future date or "now".

2. Setting a user’s assignment completion date


12.5.2 Changes that require reparsing

Once the initial parsing cycle is complete, any changes to the exit code or PBS Professional custom resource information will require the parsed data to be discarded, and all accounting log data to be reparsed. For more information see Chapter 20, "Reparsing the accounting logs", on page 323.

Any changes to the custom group information, other than those described above, will require the parsed data to be discarded, and all accounting log data to be reparsed.

12.5.3 Node configuration cron job

If your site changes their PBS Professional complex configuration often, you may want to cre-ate a cron job to automatically issue the PBS Professional pbsnodes -av command, fol-lowed by the pbsa-config-nodes command to configure PBSA with the new PBS Professional node configuration.



The pbsa-config-nodes command returns an exit code of zero (0) upon successful com-pletion. Otherwise, the command return a non-zero exit code. This exit code is not returned as standard output, but can be queried when using a cron job to wrap the pbsa-config-nodes.


Chapter 13

Adding a PBS ComplexFollow these directions to add a new PBS complex to PBSA when a site has already parsed PBS Professional accounting log files.

For example, a site has a PBS cluster that contains multiple PBS complexes:

• Complex A

• Complex B

• Complex C

Data collectors have already been installed for each complex, and accounting logs specific to each complex have been parsed. Follow these steps to add an additional PBS complex - Com-plex D.

1. Stop the PBSA service and all Data Collector services.

2. Install the Data Collector on the new PBS Complex (Complex D). Specify the host-name and port number of the machine where the PBSA Web Application and Parser are installed.

3. Optionally, build a node configuration map for the new Complex - see Chapter 9, "Supplying PBS node information", on page 113.

4. Run the parsing cycle again by first starting the PBSA service, followed by the Data Collector services.


Chapter 13 Adding a PBS Complex


Chapter 14

PBSA service commandsTwo services are responsible for running PBSA:

• The PBSA service is responsible for running the Parser, Web Application, and database.

• The Data Collector service is responsible for running the Data Collectors.

14.1 PBSA service commands

The following sections describe the commands for starting, stopping, restarting, and deter-mining the status of the PBSA service. Any PBS Professional accounting logs that have not been parsed will be processed when the PBSA service is started or restarted. The Web Appli-cation will not be available until parsing is complete.

14.1.1 Starting the PBSA service

To start the PBSA service, as the root user enter the following command at the Linux com-mand line prompt. This command will not start the PBSA Data Collector service.



Chapter 14 PBSA service commands

If the service is started successfully, messages similar to the following are displayed:

If the service does not start successfully, see Chapter 17, "Troubleshooting", on page 227.

14.1.2 Stopping the PBSA service

To stop the PBSA service, as the root user enter the following command at the Linux com-mand line prompt. This command will not stop the PBSA Data Collector service.

/etc/init.d/pbsworks-pbsa stop

If the service is stopped successfully, messages similar to the following are displayed:

Starting MonetDB server

MonetDB server started, pid: 15328 [OK]

Starting PBSWorks Login server

Platform : x86_64

64 Bit machine

Warning: Free memory is very low. Minimum memory for Base License type setup is 8 GB

INFO: Detected License Type "Base", So setting Java Max Memory to 4096m

Using CATALINA_BASE: /opt/altair/pbsa/13.0/portal/thirdparty/apache/tomcat

Using CATALINA_HOME: /opt/altair/pbsa/13.0/portal/thirdparty/apache/tomcat

Using CATALINA_TMPDIR: /opt/altair/pbsa/13.0portal/thirdparty/apache/tomcat/temp

Using JRE_HOME: /opt/altair/pbsa/13.0/portal/thirdparty/java

Using CLASSPATH: /opt/altair/pbsa/13.0/portal/thirdparty/apache/tomcat/bin/boot-strap.jar

PBSWorks Login server started, pid: 15400 [OK]

PBS Analytics watcher started successfully [OK]

PBS Analytics watcher stopped successfully [OK]

Stopping PBSWorks Login server PID 15400: Waiting to shutdown, please wait.

Waiting to shutdown, please wait.

Successfully stopped PBSWorks Login server

[OK]

Stopping MonetDB server PID 15328

Successfully stopped MonetDB server [OK]


PBSA service commands Chapter 14

14.1.3 Restarting the PBSA service

To restart the PBSA service, as the root user enter the following command at the Linux com-mand line prompt. This command will not restart the PBSA Data Collector service.

/etc/init.d/pbsworks-pbsa restart

If the service is restarted successfully, messages similar to the following are displayed:

If the service does not restart successfully, see Chapter 17, "Troubleshooting", on page 227.

Restarting PBSWorks Login server

PBS Analytics watcher stopped successfully [OK]

Stopping PBSWorks Login server PID 16121: Successfully stopped PBSWorks Login server

[OK]

Stopping MonetDB server PID 16049

Successfully stopped MonetDB server [OK]

Starting MonetDB server

MonetDB server started, pid: 16287 [OK]

Starting PBSWorks Login server

Platform : x86_64

64 Bit machine

Warning: Free memory is very low. Minimum memory for Base License type setup is 8 GB

INFO: Detected License Type "Base", So setting Java Max Memory to 4096m

Using CATALINA_BASE: /opt/altair/pbsa/13.0/portal/thirdparty/apache/tomcat

Using CATALINA_HOME: /opt/altair/pbsa/13.0/portal/thirdparty/apache/tomcat

Using CATALINA_TMPDIR: /opt/altair/pbsa/13.0/portal/thirdparty/apache/tomcat/temp

Using JRE_HOME: /opt/altair/pbsa/13.0/portal/thirdparty/java

Using CLASSPATH: /opt/altair/pbsa/13.0/portal/thirdparty/apache/tomcat/bin/boot-strap.jar

PBSWorks Login server started, pid: 16359 [OK]

PBS Analytics watcher started successfully [OK]



14.1.4 Determining the status of the PBSA service

To determine the status of the PBSA service, as the root user enter the following command at the Linux command line prompt:

/etc/init.d/pbsworks-pbsa status

Messages similar to the following will be displayed:

14.2 Data collector service commands

The following sections describe the commands for starting, stopping, restarting, and deter-mining the status of the PBSA Data Collector service. These commands must be executed on each machine where a Data Collector is installed.

14.2.1 Starting the PBSA Data Collector

To start the PBSA Data Collector service, as the root user enter the following command at the Linux command line prompt. This command will not start the PBSA service.


If the service is started successfully, messages similar to the following are displayed:

If the service does not start successfully, see Chapter 17, "Troubleshooting", on page 227.

14.2.2 Stopping the PBSA Data Collector service

To stop the PBSA Data Collector service, as the root user enter the following command at the Linux command line prompt. This command will not stop the PBSA service.

/etc/init.d/pbsworks-dc stop

If the service is stopped successfully, messages similar to the following are displayed:

PBSWorks Login server (pid 18429) is running

Starting PBSWorks Data Collector

PBSWorks Data Collector started, pid: 23615 [OK]

Stopping PBSWorks Data Collector PID 22258: [OK]


PBSA service commands Chapter 14

14.2.3 Restarting the PBSA Data Collector service

To restart the PBSA Data Collector service, as the root user enter the following command at the Linux command line prompt. This command will not restart the PBSA service.

/etc/init.d/pbsworks-dc restart

If the service is restarted successfully, messages similar to the following are displayed:

If the service does not restart successfully, see Chapter 17, "Troubleshooting", on page 227.

14.2.4 Determining the status of the PBSA Data Collector service

To determine the status of the PBSA Data Collector service, as the root user enter the follow-ing command at the Linux command line prompt:

/etc/init.d/pbsworks-dc status

Messages similar to the following will be displayed:

Restarting PBSWorks Data Collector

Stopping PBSWorks Data Collector PID 19607: [OK]

Starting PBSWorks Data Collector

PBSWorks Data Collector started, pid: 20171 [OK]

PBSWorks Data Collector (pid 20723) is running


Chapter 15

PBSA commandsWith PBSA 13.0, supplying the following site specific information is optional, but improves PBSA’s ability to collect, analyze and display information about your HPC cluster.


• holidays


• exit codes

• custom groups


When PBS Professional node information is not supplied, the data displayed in the PBSA charts is confined to the cluster’s job activity (extracted from the PBS Professional accounting logs). You may supply node information at any time and once the information is available PBSA can produce richer, more meaningful charts. For a more complete explanation see “Optional Node Configuration Improves Installation Process” on page xi.

PBSA provides the following defaults for holidays, prime/non-prime time hours, and custom groups for the last five years. It also supplies defaults for exit codes. You may use these defaults or provide site specific information.

Similar to the node configuration, PBSA 13.0 now allows holiday, prime time, and non-prime time information to be changed at any time and the PBSA charts will reflect those changes without having to reparse the accounting logs. For more information see “Reparsing Not Required for Holiday, Prime, and Non-Prime Time Changes” on page xiv


Chapter 15 PBSA commands

The following sections provide information about commands that can be used to configure and administer PBSA:

• Analyzing a PBS Professional complex

• Configuring nodes

• Configuring holidays

• Configuring custom groups

• Configuring exit codes

• Configuring the Parser

• Resetting the PBSA dataset

• Changing the PBSA database password

Table 15-1: Configuration defaults

Configuration Year(s) Default

Holiday 2011-2015 New Year’s Day

Prime

Non-prime time

2011-2015 Monday - Friday

Saturday

Sunday

Prime time begins at 09:00 am

Non-prime time begins at 5:30 pm



Exit codes N/A Successful

Unsuccessful

0

Refer to section 18.2.3, “Default exit code configuration data”, on page 266.

Custom group 2011-2015 “OTHERS” Required by PBSA - do NOT delete


PBSA commands Chapter 15

15.1 Analyzing a PBS Professional complex

Use the pbsa-setup-counsellor command to analyze PBS Professional complex information before installing and configuring PBS Analytics.

This command must be executed as the root user on the machine where the Data Collector is deployed, and must be run for each PBS Professional complex.

This command will inspect the accounting logs for a PBS Professional complex, and return the following information by year:

• PBS Professional server name

• PBS Professional server as specified in the accounting logs

• Furnishing servers if peer scheduling is configured

• Nodes found in the accounting log but are missing from the pbsnodes -av output

• Offline or down nodes that require review

• Unused nodes

• Checkpointing is enabled for this PBS Professional complex

In addition, this command provides the following information for the PBS Professional com-plex:

• An estimate of licensing records

• The date of the oldest job found in the accounting logs

• Fully qualified domain name of the machine where the Data Collector will be installed

• Approximate disk space necessary to store the accounting log data in the PBSA database

• Custom resources defined for the PBS Professional complex

15.1.1 Usage

pbsa-setup-counsellor --use-default-setting={true | false} [options]



Table 15-1: Command options and arguments

Option Argument Description

--version n/a Use this option to display the command's version number. This option can-not be used in conjunction with any other options.

--help n/a Use this option to display help for the command. This option cannot be used in conjunction with any other options.

--use-default-setting=

DFLT_SETTING

where DFLT_SETTING is either “true” or “false”.

Mandatory.

If this value is set to “false”, then the following options must be specified:

--accounting-directory

--pbs-nodes

--pbs-config

--scheduler-config

Otherwise, if set to “true” these same options default to their respective default values.

--accounting-directory=

ACCT_DIR

where ACCT_DIR is the directory path to the location of the PBS Professional accounting logs.

Defaults to PBS_HOME/server_priv/accounting.

Use this option to specify a non-default location of the PBS Professional account-ing log directory.



--pbs-nodes=

PBSNODES_AV

where PBSNODES_AV is the filename of the pbsnodes -av output file.

When this option is not specified, the pbsa-setup-counsellor command will run the pbsnodes -av command, and its output will be written to a file in the directory where the pbsa-setup-counsellor command is executed.

Use this option to specify the pbsnodes -av out-put file.

--pbs-config=

PBS_CONFIG

where PBS_CONFIG is the directory path to the location of the PBS Professional con-figuration file.

Defaults to

/etc/pbs.conf.

Use this option to specify a non-default location of the PBS Professional configu-ration file.

--scheduler-config=

SCHED_CONFIG

where SCHED_CONFIG is the directory path to the location of the PBS Profes-sional scheduler configura-tion file.

Defaults to PBS_HOME/sched_priv/sched_config.

Use this option to specify a non-default location of the PBS Professional sched-uler configuration file.

--resource-def=

RESOURCE_DEF

where RESOURCE_DEF is the directory path to the location of the PBS Profes-sional resource definition file.

Defaults to PBS_HOME/server_priv/resourcedef.

Use this option to specify a non-default location of the PBS Professional resource definition file.





15.1.2 Examples

15.1.2.1 Analyze a PBS Professional complex using defaults

Executing this command will inspect the PBS Professional accounting logs on the machine where the command is executed. The following default values will be used to determine the location of the following PBS Professional files:

• Accounting logs location = PBS_HOME/server_priv/accounting

• Configuration file = /etc/pbs.conf

• Scheduler configuration file = PBS_HOME/sched_priv/sched_config

In addition, the command will run the pbsnodes -av command, and its output will be written to a file in the directory where the pbsa-setup-counsellor command is exe-cuted.

pbsa-setup-counsellor --use-default-setting=true

15.1.2.2 Analyze a PBS Professional complex using non-defaults

Executing this command will inspect the PBS Professional accounting logs on the machine where the command is executed. It will use the specified options to determine the location of the following PBS Professional files:

• Accounting logs location

• pbsnodes -av output file

• Configuration file

• Scheduler configuration file

pbsa-setup-counsellor --use-default-setting=false --accounting-directory=/path/acct_logs --pbs-nodes=/path/pbs_nodes_av_file --pbs-config=/path/pbs_conf --scheduler-config=/path/scheduler_config

15.1.2.3 Specify a non-default resource definition file

Executing this command will inspect the PBS Professional accounting logs on the machine where the command is executed. It specifies a non-default location of the PBS Professional resource definition file.

pbsa-setup-counsellor --use-default-setting=true --resource-def=/path/resource_def



15.1.3 Error Messages

Table 15-2: Command error messages

Error MessageError Description and

Troubleshooting

Python version XX is not supported. Sup-ported version is 2.4.x or higher.

Please update the Python version to 2.4 or higher.

Error: Connection refused XXXX: cannot connect to server XXXX, error=XXXX

Please make sure PBS Pro server is run-ning.

Cannot connect to the PBS Professional Server. Please verify that PBS Professional is up and running.

ERROR:PBS Pro configuration file XXXX is not correct.

Please verify that the PBS Professional con-figuration file specified is correct. The val-ues for PBS_HOME and PBS_EXEC could not be found.

Instance of PBS Pro not found. Please make sure you are running this script on PBS Pro server

An instance of PBS Professional Server is not installed on this machine. Verify that you are running this command on a machine that hosts the PBS Professional Server.

ERROR: Invalid directory path XXXX. The specified directory path does not exist or is not a directory.

ERROR: Invalid File path XXXX. The specified file does not exist or is not a file.



15.2 Configuring nodes

If your site implements peer scheduling see Chapter 22, "Peer Scheduling", on page 333 for additional information on how to deploy and configure PBSA.

Use the pbsa-config-nodes command to configure PBS Professional node information.

This command must be executed as the root user on the machine where the Web Application and Parser are installed.

The following information is mandatory input for the pbsa-config-nodes command, and can be obtained by running the pbsa-setup-counsellor command:

• the PBS Professional server hostname as specified in the PBS Professional accounting logs

• the fully qualified domain name of the machine where the Data Collector is installed

• the output from the PBS Professional command: pbsnodes -av.

The node detail information is stored by year in the file nodedetails_YYYY.conf where "YYYY" is the four digit year. For more information about this file see section 18.5, “Node details file”, on page 275. By default, the output of this command is written to:


The pbsa-config-nodes command returns an exit code of zero (0) upon successful com-pletion. Otherwise, the command return a non-zero exit code. This exit code is not returned as standard output, but can be queried when using a cron job to wrap the pbsa-config-nodes command.

Use this command to update PBSA with current node information when the following events occur:

• initial installation of PBSA

• addition of new nodes

• deletion of nodes

• nodes that are taken offline

• nodes that have been brought back online

Once the accounting logs are parsed for the first time, any changes to the PBSA node informa-tion will be incorporated in the next parsing cycle. If you do not want to wait until the next parsing cycle to include your changes, you may restart the PBSA service to initiate a parsing cycle.



15.2.1 Usage

pbsa-config-nodes PRIMARY_SERVER SECONDARY_SERVER DC_FQDN PATH_TO_PBSNODES.OUT [options]

where:

PRIMARY_SERVER is the name of the PBS Professional primary server as specified in the PBS Professional accounting logs

SECONDARY_SERVER (optional) is the name of the PBS Professional failover server

DC_FQDN is the fully qualified domain name of the machine where the Data Collector is installed.

PATH_TO_PBSNODES.OUT is the output file from the command: pbsnodes -av

A fully qualified domain name (FQDN),is a domain name that specifies its exact location in the tree hierarchy of the Domain Name System (DNS). It specifies all domain levels, includ-ing the top-level domain and the root zone.

For example, given a device with a local hostname myhost and a parent domain name example.com, the fully qualified domain name is myhost.example.com. The FQDN therefore uniquely identifies the device.

The FQDN can be obtained by issuing the following Unix commands:

hostname -f or hostname --fqdn

The PRIMARY_SERVER must be the server name as specified in the PBS accounting logs.



If any nodes are offline or down, the pbsa-config-nodes command will return a mes-sage similar to the following:

This message will be returned for nodes having a comment equal to “node down:communica-tion closed”, or having the following states:

• down,unresolvable

• state-unknown,offline

• state-unknown,down

The system administrator must perform some analysis at this point to determine the cause of the downed node. For instance, there may be a communication problem between the PBS Professional MOM and headnode that is causing the node to be marked as down, when actu-ally the node is up and running. In this case correct the communication problem, verify that node resources are correct, re-run the pbsnodes -av command, and issue the pbsa-config-nodes command again.

If the node is down for a valid reason, such as maintenance, then you will have to manually edit the pbsnodes.out file. You may:

• change the state of the node to an online state such as “free”. If you choose to set the state of a downed node to “free”, verify that the following two parameters are present in the pbsnodes -av output for that node:

• resources_available.ncpus

• resources_available.mem

If these parameters are not present, then manually enter this information for that node.

• delete the node from the pbsnodes.out file. You can configure PBSA at a later date when the node comes online (see section 12.5, “Configuration changes after initial pars-ing”, on page 137). However, deleting the node from the file creates a situation where PBSA is “unaware” of this node during parsing of the accounting logs. PBSA cannot account for those accounting log entries specific to that node, therefore PBSA will write these log entries to a file in a recovery directory.

Once all manual editing is complete, issue the pbsa-config-nodes command again.

Please check the pbsanodes-av output file, some of the nodes are offline. Following nodes are offline: ['tribm3550m3-02']







--start-date=STARTDATE where STARTDATE is in the format of YYYY-MM-DD. Default is the current date.

Use this option to define the start date for which this PBS Professional server configuration is in effect.

--end-date=ENDDATE where ENDDATE is in the format of YYYY-MM-DD. Default is “now” which denotes that this configura-tion is currently in effect.

Use this option to define the end date for which this PBS Professional server configuration is in effect.

--furnishing-servers= FUR_SERVERS

Comma separated list of fur-nishing servers.

Run the pbsa-setup-counsellor command to obtain a list of furnishing servers for this PBS Profes-sional complex.

Use this option to specify a list of furnishing servers when peering is config-ured. See Chapter 22, "Peer Scheduling", on page 333 for more infor-mation.



--config-dir=CONFIGDIR where CONFIGDIR is the directory path to the location where the output of this command will be directed. The default PBSA configu-ration directory is INSTALL_DIR_13.0/portal/services/pbsa/parser/mod-ules/pbsdata/con-fig.

Use this option to specify an alternate configuration directory.

--show n/a Use this option to display the current configuration on the screen for review. Use this option in conjunc-tion with the --start-date option to view a configura-tion from a previous year. Default is to display the current year’s configura-tion information.

--new n/a Use this option to over-write any existing node details configuration.





15.2.2 Examples

All examples assume that the PBS Professional command pbsnodes -av has been exe-cuted and the output from this command has been written to a file called pbsnodes.out

15.2.2.1 Configure current node details

Execute the following command if you want to load node information that is effective today and will continue to be in effect going forward (start date defaults to today and end date defaults to “now”).

pbsa-config-nodes hwesuse113-01 hwesuse113-01.ms.com pbsnodes.out

15.2.2.2 Configure previous year’s node details

Execute the following command if you want to load node configuration information for a pre-vious year:

pbsa-config-nodes hwesuse113-01 hwesuse113-01.ms.com pbsnodes.out --start-date=2014-01-01 --end-date=2014-12-31

15.2.2.3 Configure node details for a site implementing peer scheduling

This example shows how to configure node details for a site that has implemented peer sched-uling. Peer scheduling requires a list of furnishing servers - see Chapter 22, "Peer Schedul-ing", on page 333 for more information.

pbsa-config-nodes Bangalore bangalore.ms.com pbsnodes.out --furnishing-servers=Troy,MV

15.2.2.4 Redirect the output of the command

This example shows how the output of a command can be directed to an alternate location. This allows the output of the command to be saved without overwriting the contents of the default configuration directory.

pbsa-config-nodes hwesuse113-01 hwesuse113-01.ms.com pbsnodes.out --config-dir=/tmp/pbsa-config-dir



15.2.2.5 Refresh the node configuration

Performing this command will overwrite the current node configuration. Use this command when you need to discard the current node configurations and start afresh.

pbsa-config-nodes hwesuse113-01 hwesuse113-01.ms.com pbsnodes.out --new

15.2.2.6 Show previous year’s configuration

This command shows the node configuration for a previous year.

pbsa-config-nodes --start-date=2014-01-01 --show






Troubleshooting

The PBSWORKS_COMMAND_HOME or PBSWORKS_SERVER_HOME vari-able was not set, and I was unable to find or open a pbsworks-pbsa.conf file.

The environment variables PBSWORKS_COMMAND_HOME and PBSWORKS_SERVER_HOME have not been exported to the environment. Contact your systems administrator to set up the environment properly.

Error occurred while setting PBSWORKS_COMMAND_HOME or PBSWORKS_SERVER_HOME.




Error occurred while get uid for user XXXX.

Unable to get user id for current user. Con-tact your system administrator to report this error.

Operation not permitted, Please run com-mand as XXXX user or ROOT.

The user does not have permissions to run this command. Please run the command as the PBSA service user specified during installation or as root user.

memory unit XXXX is not supported An error occurred while processing the pbsnodes- av output during the conver-sion of memory units to GB. The memory unit did not equal one of the following valid options: b, kb, mb, gb, tb, pb.

The configuration file is empty. This error may occur after executing this command with the --show option. The con-figuration file is empty, therefore nothing has been configured yet.



Invalid furnishing server names FUR_SERVERS must be a comma separated list of furnishing servers.

Please check the pbsnodes -av output file, some of the nodes are offline. Following nodes are offline: [‘xxxxxxxxxx’]

Please investigate the reason for the downed node and either correct the issue within PBS Professional, or if this is a planned outage, then manually edit the pbsnodes -av output file and modify the state of the node to an online status, or delete the node.

Incorrect number of arguments. Try pbsa-config-nodes --help.

Please review the usage information for this command. You can obtain the usage infor-mation by using the --help option.

Config directory does not exists.

Path XXXX.

The PBSA configuration directory INSTALL_DIR_13.0/portal/ser-vices/pbsa/parser/modules/pbsdata/config does not exist. An error may have occurred during installation of PBSA. Please re-install the product.

Unable to instantiate the PBSA ToolKit. Please check your PBSWORKS_-SERVER_HOME variable.

The environment variable PBSWORKS_-SERVER_HOME may not have been exported to the environment. Contact your systems administrator to set up the environ-ment properly.

Unable to read the "pbsnodes -av" output file for user XXXX.

Path: XXXX.

The pbsnodes -av output file does not exist.

Unable to read the "pbsnodes -av" output file for user XXXX.

Please check file permissions.

Incorrect file permissions for opening the pbsnodes -av output file for read access.



Troubleshooting



Unable to open the “nodedetails_<year>.conf” file.

Please check the existence and permissions of the nodedetails_YYYY.conf file located at INSTALL_DIR_13.0/por-tal/services/pbsa/parser/mod-ules/pbsdata/config.

The startdate XXXX provided is not valid. STARTDATE must be in the format of YYYY-MM-DD.

The enddate XXXX provided is not valid. ENDDATE must be in the format of YYYY-MM-DD or must be “now”.

The startdate XXXX provided must not be greater than end date XXXX.

Invalid date range. STARTDATE must be less than ENDDATE.

The startdate year XXXX and enddate year XXXX must be the same.

Specified date range must not span over sev-eral years.

Start date format is incorrect. Expected is YYYY-MM-DD.

Dates should be in the format of YYYY-MM-DD.



Troubleshooting



15.3 Configuring holidays

Use the pbsa-config-holidays command to configure the yearly holiday schedule, and to configure prime (normal working hours) and non-prime (non-working hours) hours per day of week.


Mandatory input required for this command is the name of the holiday or the day of the week.

The holiday configuration information is stored by year in the file holiday_YYYY.conf, where "YYYY" is the four digit year. By default the output of this command is written to:


For more information about this file see section 18.1, “Holiday configuration file”, on page 255.

By default, PBSA is configured with a single holiday, New Year’s day, for the years 2011 through 2015. For Monday through Friday, prime times hours begin at 09:00 am, and non-prime time hours begin at 5:30 pm. For Saturday and Sunday, the entire day is scheduled as non-prime time.

Once the accounting logs are parsed for the first time, any changes to the PBSA holiday infor-mation will be incorporated in the next parsing cycle. If you do not want to wait until the next parsing cycle to include your changes, you may restart the PBSA service to initiate a parsing cycle.



15.3.1 Usage

pbsa-config-holidays {HOLIDAY|DAY_OF_WEEK} [options]

where:

HOLIDAY is the name of the holiday. It can contain alphanumeric characters and/or underscores.

DAY_OF_WEEK is the name of the day of week. Valid values:

• weekdays (i.e., specifies any week day that has not already been defined sepa-rately - cannot use this value if Monday-Friday have already been con-figured separately)

• weekends (Saturday and Sunday - cannot be used if Saturday and Sunday have already been configured separately)

• monday

• tuesday

• wednesday

• thursday

• friday

• saturday

• sunday



--version n/a Use this option to display the com-mand's version number. This option cannot be used in conjunction with any other options.





Use this option to define the start date of the holiday. This option can also be used to configure prime/non-prime time hours for a pre-vious year’s config-uration.

--end-date=ENDDATE where ENDDATE is in the format of YYYY-MM-DD. Default is the current date.

Use this option to define the end date of the holiday.





--prime-time=PRIMETIME where PRIMETIME is in the format of HH:MM in the 24 hour format, or may be “all” which denotes that all hours are prime time hours, or may be “none” which denotes that all hours are non-primetime hours.

If PRIMETIME is set to “all”, then NONPRIME-TIME should be set to “none”. If PRIMETIME is set to “none”, then NONPRIMETIME should be set to “all”. Default is 09:00.

Use this option to define the prime-time start time.

--non-prime-time=NONPRIMETIME where NONPRIME-TIME is in the format of HH:MM in the 24 hour format, or may be “all” which denotes that all hours are non-primetime hours, or “none” which denotes that all hours are prime time hours.

If NONPRIMETIME is set to “all”, then PRIME-TIME should be set to “none”. If NONPRIME-TIME is set to “none”, then PRIMETIME should be set to “all”. Default is 17:30.

Use this option to define the non-primetime start time.





--config-dir=CONFIGDIR where CONFIGDIR is the directory path to the location where the out-put of this command will be directed. The default PBSA configuration directory is INSTALL_DIR_13.0/portal/ services/pbsa/parser/modules/pbsdata/config.

Use this option to specify an alternate configuration direc-tory.

--show n/a. Use this option to display the current configuration on the screen for review. Use this option in conjunction with the --start-date option to view a configuration from a previous year. Default is to dis-play the current year’s configura-tion information..

--new n/a Use this option to overwrite any exist-ing holiday configu-ration.





15.3.2 Examples

15.3.2.1 Configure a one-day holiday

This example shows how to configure a holiday for a single day.

pbsa-config-holidays MemorialDay --start-date=2015-05-25 --end-date=2015-05-25

15.3.2.2 Configure a multiple-day holiday

This example shows how to configure a holiday for multiple days.


15.3.2.3 Configure prime/non-prime hours for a day of the week

Executing this command configures prime time to start at 9:00 am and non-prime to start at 6:00 pm for Monday.

pbsa-config-holidays monday --prime-time=09:00 --non-prime-time=18:00

--remove n/a Use this option to remove a holiday or day of the week from the current holiday configura-tion. AHOLIDAY or DAY_OF_WEEK must be designated to uniquely identify the holiday or day of the week to be removed.





15.3.2.4 Configure prime/non-prime hours using defaults

Executing this command configures prime time to start at 8:00 am and non-prime time to start at 5:30 pm for Monday. Prime time start defaults to 8:00 am and non-prime start defaults to 5:30 pm (17:30).

pbsa-config-holidays monday

15.3.2.5 Configure prime/non-prime hours for weekends

Executing this command configures prime time to start at 9:00 am and non-prime time to start at 6:00 pm for weekends (Saturday and Sunday).

pbsa-config-holidays weekends --prime-time=09:00 --non-prime-time=18:00

15.3.2.6 Configure prime/non-prime hours using weekdays

Executing this command configures prime/non-prime time hours for any weekdays (Monday-Friday) not specified separately. For instance, if prime time and non-prime time have already been specified for Monday and Tuesday, then this command will specify prime time and non-prime time for days Wednesday through Friday.

pbsa-config-holidays weekdays --prime-time=09:00 --non-prime-time=18:00

15.3.2.7 Configure prime/non-prime hours for previous year

Executing this command configures prime time to start at 9:00 am and non-prime time to start at 6:00 pm on every Monday for the year 2014.

pbsa-config-holidays monday --start-date=2014-01-01 --prime-time=09:00 --non-prime-time=18:00



pbsa-config-holidays MemorialDay --start-date=2015-05-25 --end-date=2015-05-25 --config-dir=/tmp/pbsa-config-dir



15.3.2.9 Refresh the holiday configuration

Executing this command will overwrite the current holiday configuration. Use this command when you need to discard the current holiday configuration and start afresh.

pbsa-config-holidays MemorialDay --start-date=2015-05-25 --end-date=2015-05-25 --new

15.3.2.10 Remove a holiday from the holiday configuration

Executing this command will remove a holiday from the current holiday configuration.

pbsa-config-holidays MemorialDay --remove

15.3.2.11 Remove a day from the holiday configuration

Executing this command will remove a day of the week from the current holiday configura-tion.

pbsa-config-holidays monday --remove

15.3.2.12 Show previous year’s configuration

This command shows the holiday configuration for a previous year.

pbsa-config-holidays --start-date=2014-01-01 --show




Troubleshooting


The environment variables PBSWORKS_COMMAND_HOME and PBSWORKS_SERVER_HOME have not been exported to the environment. Contact your systems administrator to set up the envi-ronment properly.








Unable to get user id for current user. Con-tact your system administrator to report this error..



Either Holiday or prime and non-prime config can be specified at a time.

Configuring a holiday or prime/non-prime cannot be done with a single command. First issue the command to configure the holidays, and then issue a second command to config-ure the prime/non-prime time hours.

The configuration file is empty. This error may occur after executing this command with the --show option. The con-figuration file is empty, therefore nothing has been configured yet.

Invalid holiday name. Holiday name must contain only any alphanumeric character and the underscore.

Please enter a valid holiday name which con-tains only alphanumeric characters and/or the underscore character.

Invalid option --remove in the command. Cannot specify the --remove option when using the --new option.



Troubleshooting



Please define all weekdays (Mon-Fri) sep-arately as "weekdays" value is removed.

The default entry for “weekdays” has been removed therefore, you must specify prime/non-prime time hours for Monday through Friday separately. For instance, you may reconfigure the prime and non-prime time hours by issuing the following command for each day of the week (Monday-Friday): pbsa-config-holidays dayofWeek --prime-time=HH:MM --non-prime-time=HH:MM where dayofWeek is: monday, tuesday, wednesday, thursday, fri-day

Please define both weekends (Sat-Sun) separately as "weekends" value is removed..

The default entry for “weekends” has been removed therefore, you must specify prime/non-prime time hours for Saturday and Sun-day separately. For instance, you may recon-figure the prime and non-prime time hours by issuing the following com-mands: pbsa-config-holidays Saturday --prime-time=HH:MM --non-prime-time=HH:MM pbsa-config-holi-days Sunday --prime-time=HH:MM --non-prime-time=HH:MM

Please make sure both weekend days (Sat & Sun) are defined.

Warning message that requests that prime/non-prime time hours be configured for both Saturday and Sunday.

Incorrect number of arguments. Try pbsa-config-holidays --help.




Troubleshooting



Config directory does not exists. The PBSA configuration directory INSTALL_DIR_13.0/portal/ser-vices/pbsa/parser/modules/pbsdata/config does not exist. An error may have occurred during installation of PBSA. Please re-install the product.



Unable to open the “holiday_<year>.conf” file.

Please check the existence and permissions of the holiday_YYYY.conf file located at INSTALL_DIR_13.0/portal/ser-vices/pbsa/parser/modules/pbsdata/config

The startdate XXXX provided is not valid.

STARTDATE must be in the format of YYYY-MM-DD.

The enddate XXXX provided is not valid. ENDDATE must be in the format of YYYY-MM-DD.







The primetime XXXX provided is not valid.

PRIMETIME must be in the format of HH:MM in the 24 hour clock.

The nonprimetime XXXX provided is not valid.

NONPRIMETIME must be in the format of HH:MM in the 24 hour clock.



Troubleshooting



15.4 Configuring custom groups

Custom groups can be used to track utilization at a project level for sites running versions of PBS Professional prior to 11.2.

The preferred method for tracking utilization at a project level is the use of the -P option when submitting a job via the qub command (see section 3.2.3, “Utilizing the qsub -P option”, on page 15).

However, for older versions of PBS, use the pbsa-config-groups command.


Mandatory input required for this command is a userid, and the custom group (i.e., project) assigned to the user. Users may be assigned to multiple groups.

The custom group information is stored by year in the file custom_group_YYYY.conf, where "YYYY" is the four digit year. By default the output of this command is written to:


For more information about this file see section 18.3, “Custom group configuration file”, on page 267.

PBSA is configured with a default group “OTHERS” for the years 2011 through 2015.

Custom groups is an alternative offered by PBSA that has been made available for legacy purposes only, and may be deprecated in future releases of PBSA.

Do NOT delete the default custom group “OTHERS”.



There are two situations that are not supported by the pbsa-config-groups command, and will require that the custom group configuration file to be manually edited (see section 18.3.5, “Situations that require editing the Custom Group Configuration file”, on page 272 for examples).

1. First, the pbsa-config-groups command does not support a user being assigned to the same project more than once in a year. For instance, a user cannot be assigned to projectA from January through March in 2015, and then again from September through December. If this situation occurs, then the configuration file must be manu-ally edited to make this assignment.

2. Secondly, since a user can be assigned to multiple projects during the same time period, PBSA allows a default custom group (project) to be selected. For instance, user1 is assigned to projectA and projectB, and projectA is designated as the “default” project. If user1 submits a job without designating the project (using the qsub -P option), then PBSA will track the job’s utilization against projectA since it was desig-nated the “default” project. The pbsa-config-groups command does not cur-rently support setting the default project. Again, the configuration file must be manually edited to make this assignment.

The date a user begins working in a group will determine which yearly configuration file is updated. If a user works in a group over consecutive years, entries must be made in multiple configuration files - one for each year.

For example, if a user is assigned to a group for the year 2014, and then continues working for that same group into 2015, then an entry for 2014 must be made in the 2014 configuration file with an ending date of December 31, 2014. A second entry must be made in the 2015 config-uration file with a start date of January 1, 2015.

If a user works in a group over consecutive years, entries must be made in multiple configuration files - one for each year



Once the accounting logs are parsed for the first time, modifications to the current year's information should only be done in two situations and do not require any reparsing of accounting logs.

1. A user is assigned to a new project.Use the pbsa-config-groups command to assign the user to the new project. Set the start date to today's date or later. The end date can be set to a future date or "now".

2. A user completes an assignment.


These custom group changes will be incorporated in the next parsing cycle. If you do not want to wait until the next parsing cycle to include your changes, you may restart the PBSA service to initiate a parsing cycle.

If the custom group information is modified for any other reason and you want this informa-tion reflected in the data that had already been parsed and loaded into PBSA, then the accounting logs will have to be reparsed.

15.4.1 Usage

pbsa-config-groups USER GROUP [options]

where:

USER is the userid

GROUP is the custom group or project to which the user is assigned.



--version n/a Use this option to display the command's version number. This option cannot be used in conjunction with any other options.





Use this option to define the date on which the user began working on the project.

--end-date=ENDDATE where ENDDATE is in the format of YYYY-MM-DD. Default is “now” which denotes that the user is still currently working in the group.

Use this option to define the date on which the user stopped working on the project.

--config-dir=CONFIGDIR where CONFIGDIR is the directory path to the location where the out-put of this command will be directed. The default PBSA configuration directory is INSTALL_DIR_13.0/portal/services/pbsa/parser/ modules/pbsdata/config.


--show n/a Use this option to display the current configuration on the screen for review. Use this option in conjunction with the --start-date option to view a con-figuration from a previous year. Default is to display the current year’s configuration informa-tion.





15.4.2 Examples

15.4.2.1 Assigning a user to a custom group

Executing this command assigns a user (tsmith) to a custom group (CFD) from January 22, 2015 to November 23, 2015.


Executing this command configures a user to be assigned to a custom group with a start date of today’s date and an end date of “now” denoting that the user is currently assigned to that custom group.

pbsa-config-groups tsmith CFD

Executing this command assigns a user (tsmith) to a custom group (CFD) from January 1, 2015 with an end date of “now” denoting that the user is currently assigned to that custom group.

pbsa-config-groups tsmith CFD --start-date=2015-01-01

--new n/a Use this option to overwrite any existing user/group config-uration.

--remove n/a Use this option to remove the specified user/custom group from the current configura-tion. Both the USER and GROUP must be specified to uniquely identify the custom group to be removed.





15.4.2.2 Assigning a user to several custom groups

Executing these commands assigns a user (tsmith) to a custom group (CFD) from January 1, 2015 to June 1, 2015, and to another custom group (CAD) from January 1, 2015 to September 1, 2015.


pbsa-config-groups tsmith CAD --start-date=2015-01-01 --end-date=2015-09-01



pbsa-config-groups tsmith CFD --start-date=2015-01-01 --end-date=2015-06-01 --config-dir=/tmp/pbsa-config-dir

15.4.2.4 Refresh the custom group configuration

Performing this command will overwrite the current custom group configuration. Use this command when you need to discard the current custom group configuration and start afresh.

pbsa-config-groups tsmith CFD --start-date=2015-01-22 --end-date=2015-11-23 --new

15.4.2.5 Remove a custom group from the configuration

This example shows how to remove a custom group from the current custom group configura-tion.

pbsa-config-groups tsmith CFD ---remove

15.4.2.6 Show previous year’s custom group configuration

This command shows the custom group configuration for a previous year.

pbsa-config-groups --start-date=2014-01-01 --show






Troubleshooting











The configuration file is empty. This error may occur after executing this command with the --show option. The con-figuration file is empty, therefore nothing has been configured yet

Incorrect number of arguments. Try pbsa-config-groups --help.





WARN: User "system" can't be removed. The default entry “system” cannot be deleted. PBSA requires this entry.




Unable to open the “custom_group_<year>.conf” file.

Please check the existence and permissions of thecustom_group_YYYY.conf file located at INSTALL_DIR_13.0/por-tal/services/pbsa/parser/mod-ules/pbsdata/config.

The startdate XXXX provided is not valid.

STARTDATE must be in the format of YYYY-MM-DD.

The enddate XXXX provided is not valid. ENDDATE must be in the format of YYYY-MM-DD or must be “now”.









Troubleshooting



15.5 Configuring exit codes

Use the pbsa-config-exits command to configure exit codes.


Mandatory input required is a comma separated list or range of exit codes.

The exit code information is stored in the file exit_status.conf. By default the output of this command is written to:


For more information about this file see section 18.2, “Exit code configuration file”, on page 264.

PBSA is configured by default with two exit reasons “Successful” and “Unsuccessful” that are mapped to specific exit codes. An example of this default mapping can be found in sec-tion 18.2.3, “Default exit code configuration data”, on page 266.

Once the initial parsing cycle is complete, any changes to the exit code information will require the parsed data to be discarded, and all accounting log data to be reparsed. For more information see Chapter 20, "Reparsing the accounting logs", on page 323.

15.5.1 Usage

pbsa-config-exits EXIT_CODES [options]

where:

EXIT_CODES is a comma separated list of exit codes and/or a range of exit codes.

A range of negative exit codes has to be configured using a comma separated list.

The default exit code “Still-Running” must not be removed from the exit configuration.







--reason=REASON where REASON is a expla-nation or description of the exit code. Default is “Unspecified”.

Use this option to provide a description of the exit code.

--application=APPLICATION where APPLICATION is a name of an application such as “PBSJobApp”. Default is “All”.

The value assigned to --application directly corre-sponds to the application name specified via the PBS Professional "software" attribute.

Use this option to catego-rize the exit codes by application or solver. If this option is not defined, then the default is “All” applications.



--config-dir=CONFIGDIR where CONFIGDIR is the directory path to the loca-tion where the output of this command will be directed. The default PBSA configuration direc-tory is INSTALL_DIR_13.0/portal/services/pbsa/parser/ modules/pbsdata/config.

Use this option to specify an alternate configuration directory.

--show n/a Use this option to display the current configuration on the screen for review. This option cannot be used in conjunction with any other options.

--new n/a Use this option to over-write any existing exit code configuration.

--remove n/a Use this option to remove a reason/application from the current exit code con-figuration. The --reason and --application options are used to uniquely iden-tify the exit code to be removed. If these options are not specified then the reason defaults to “Unspecified” and the application defaults to “All”.





15.5.2 Examples

15.5.2.1 Configuring an exit code

Executing this command configures an exit code of “1” to be defined as “Successful” for the application PBSJobApp.

pbsa-config-exits 1 --reason=Successful --application=PBSJobApp

15.5.2.2 Configuring a comma separated list of exit codes

Executing this command configures a list of exit codes to be defined as “Unsuccessful” for the application PBSJobApp.

pbsa-config-exits 2,143,255 --reason=Unsuccessful --application=PBSJobApp

Executing this command configures a list of exit codes to be defined as “Unspecified” for all applications (defaults).

pbsa-config-exits 2,143,255

15.5.2.3 Configuring a range of exit codes

Executing this command configures a range of exit codes from 10 to 20 to be defined as “Unsuccessful” for the application PBSJobApp.

pbsa-config-exits 10-20 --reason=Unsuccessful --application=PBSJobApp

A range of negative exit codes has to be configured using a comma separated list - see section 15.5.2.5, “Configuring negative exit codes”, on page 187.

15.5.2.4 Configuring both a list and a range of exit codes

Executing this command configures both a comma separated list and a range of exit codes to be defined as “Unsuccessful” for the application PBSJobApp.

pbsa-config-exits 1,2,3,10-20 --reason=Unsuccesful --application=PBSJobApp

A range of negative exit codes has to be configured using a comma separated list - see section 15.5.2.5, “Configuring negative exit codes”, on page 187.



15.5.2.5 Configuring negative exit codes

Executing this command configures negative exit codes.

pbsa-config-exits 1-10,11,12-15,-13,-14,-15

A range of negative exit codes can be configured using a comma separated list.



pbsa-config-exits 2,143,255 --config-dir=/tmp/pbsa-config-dir

15.5.2.7 Refresh the exit code configuration

Performing this command will overwrite the current exit code configuration. Use this com-mand when you need to discard the current exit code configuration and start afresh.

pbsa-config-exits 1 --reason=Successful --application=PBSJobApp --new

15.5.2.8 Refresh the exit code configuration with default entry

Performing this command will overwrite the current exit code configuration. This command will create a new exit code configuration file with a single default entry representing a job is “Still Running”.

pbsa-config-exits --new

15.5.2.9 Remove an exit code from the exit code configuration

Performing this command will remove exit codes defined for the application “All” and reason “Unspecified” (defaults) from the current exit code configuration .

pbsa-config-exits --remove

Performing this command will remove exit code defined for the application “PBSJobApp” and reason “Unspecified” (default) from the current exit code configuration

pbsa-config-exits --application=PBSJobApp --remove

Performing this command will remove exit codes defined for the application “All” (default) and reason “Unsuccessful” from the current exit code configuration.

pbsa-config-exits --reason=Unsuccessful --remove



Performing this command will remove exit code defined for the application “PBSJobApp” and reason “Unsuccessful” from the current exit code configuration

pbsa-config-exits --application=PBSJobApp --reason=Unsuccessful --remove




Troubleshooting











The configuration file is empty. This error may occur after executing this command with the --show option. The con-figuration file is empty, therefore no exit codes have been configured yet.



WARN: User can't update private settings for exitreason: "Still Running"

The default entry “Still Running” cannot be deleted. PBSA requires this entry.


Incorrect number of arguments. Try pbsa-config-exits --help.





Unable to open the “exit_status.conf” file. Please check the existence and permissions of the exit_status.conf file located at INSTALL_DIR_13.0/portal/ser-vices/pbsa/parser/modules/pbsdata/config



Troubleshooting



15.6 Configuring the Data Collector

The PBSA Data Collector makes the PBS Professional accounting logs accessible to the PBSA Web Application. It has two functions:

• copies the accounting logs from your site’s PBS Professional accounting log location to an intermediate holding area every one (1) minute

• transports the information stored in the holding area to the machine where the PBSA Web Application and Parser are installed every one (1) minute

You can change the time interval for both these functions using the pbsa-config-dc com-mand, however this is only recommended for advanced users of PBSA. The size of your site’s PBS complex and the number and size of the accounting logs will dictate the length of time required for copying and transporting the accounting logs to the machine where the Parser is installed.

This command must be executed on the machine where the Data Collector is deployed.

By default the interval for copying the accounting logs is written to:

INSTALL_DIR_13.0/dc/config/modules.xml

By default the interval for transporting the accounting logs is written to:

INSTALL_DIR_13.0/dc/config/config.xml.

If the PBSA Data Collector is currently running, then it will need to be restarted after issuing the pbsa-config-dc command.

15.6.1 Usage

pbsa-config-dc [options]



--version n/a Use this option to dis-play the command's version number. This option cannot be used in conjunction with any other options.



--help n/a Use this option to dis-play help for the com-mand. This option cannot be used in con-junction with any other options.

--transport-interval=TRANSPORT where TRANSPORT is in minutes.

Default is 10 minutes.

Use this option to define the transport interval in minutes for transporting the PBS Professional account-ing logs to the machine where the PBSA Web Applica-tion and Parser are installed.

--collection-interval=COLLECTION where COLLECTION is in minutes.

Default is 5 minutes.

Use this option to define the collection interval in minutes for collecting the PBS Professional account-ing logs (copying them to an intermediate area prior to transporting to the machine where the Web Application and Parser are installed).





15.6.2 Examples

15.6.2.1 Configuring the Data Collector

This example shows how to configure the Data Collector to collect the PBS accounting logs and copy them to an intermediate area every 50 minutes, and then transport the log informa-tion to the machine where the PBSA Web Application and Parser are installed every 60 min-utes.

pbsa-config-dc --transport-interval=60 --collection-interval=50



pbsa-config-dc --transport-interval=60 --collection-interval=50 --config-dir=/tmp/pbsa-config-dir

--config-dir=CONFIGDIR where CONFIGDIR is the directory path to the location where the out-put of this command will be directed. The default PBSA configu-ration directory is for the Data Collector is INSTALL_DIR_13.0/dc/config.


--show n/a. Use this option to dis-play the current Data Collector configura-tion on the screen for review.








Troubleshooting

The DC_SERVER_HOME or DC_CONF_FILE variable was not set, and I was unable to find or open a pbsworks-dc.conf file.

The environment variables DC_SERVER_HOME and DC_CONF_FILE have not been exported to the environment. Contact your systems administrator to set up the environment properly.

Error occurred while setting DC_SERVER_HOME.

The environment variable DC_SERVER_HOME has not been exported to the environment. Contact your systems administrator to set up the environment properly.

Python version XX is not supported. Supported version is 2.4.x or higher.


Operation not permitted, Please run command as XXXX user.

The user does not have permissions to run this command. Please run the command as the PBSA service user specified during installation.

Option XXXX argument XXXX must be in min-utes. Try dc-config --help

Then specified option must be expressed in minutes.

Error occurred while get uid for user XXXX. Unable to get user id for current user. Contact your system administrator to report this error.

Please specify --transport-interval=TRANSPORT_INTERVAL or --collection-interval=COLLECTION_INTERVAL

You must specify either a transport interval and/or a collection interval.



The configuration file is empty. This error may occur after executing this command with the --show option. The configuration file is empty, there-fore nothing has been configured yet.

Config directory does not exists. The PBSA configuration directory INSTALL_DIR_13.0/portal/services/pbsa/parser/mod-ules/pbsdata/config does not exist. An error may have occurred during installation of PBSA. Please re-install the product.

Unable to instantiate the PBSA ToolKit. Please check your PBSWORKS_SERVER_HOME vari-able.

The environment variable PBSWORKS_SERVER_HOME may not have been exported to the envi-ronment. Contact your systems administrator to set up the environ-ment properly.

Unable to read the dc config file. Please check the existence and per-missions of the config.xml and/or modules.xml files located at INSTALL_DIR_13.0/dc/con-fig



Troubleshooting



15.7 Configuring the Parser

The PBSA Parser reads the PBS Professional accounting logs, parses the appropriate informa-tion from the accounting logs, and then loads this information into a database used by the PBSA Web Application.

By default, the Parser is started 15 minutes after the PBSA service is started. This delay gives the Data Collector time to copy the PBS accounting logs to the machine where the Parser is installed. You may reduce the length of this default, however this is only recommended for advanced users of PBSA. The size of your site’s PBS complex and the number and size of the accounting logs will dictate the length of time required for copying the accounting logs to the machine where the Parser is installed. If parsing begins before all accounting logs are trans-ported, then any unprocessed accounting log files will be processed in the next parsing cycle. Alternately, after the initial parsing cycle completes, you can restart the PBSA service to initi-ate a second parsing cycle.

Use the pbsa-config-parser to configure this delay. It is recommended to set the Parser cycle delay to no less than 10 minutes.


In addition, you can configure when the daily Parser cycle will run (currently this cycle should not be run more than once in a 24-hour period).

This information is written to

INSTALL_DIR_13.0/portal/services/pbsa/parser/modules/pbsdata/config/config.xml

The PBSA service must be restarted after issuing the pbsa-config-parser command.

The Parser delay is configured for 15 minutes by default.

By default, the Parser cycle is configured to run once a day at 1:00 A.M.



15.7.1 Usage

pbsa-config-parser [options]



--version n/a Use this option to display the command's version number. This option cannot be used in conjunction with any other options.

--help n/a Use this option to display help for the command. This option cannot be used in con-junction with any other options.

--parser-cycle-delay=DELAY where DELAY is in min-utes.

Use this option to define the Parser cycle delay in min-utes. This is the delay between when the PBSA ser-vice begins running and when the Parser begins to process the accounting log information.

--parser-timer=TIMER where TIMER is a Cron expression specified between double quotes. For example, "0 1 * 0 0 ?"

Use this option to define when the daily parsing cycle should run.



--job-duration=DURATION where DURATION is in days.

Use this option to define the maximum number of days a “S” (running) job will be considered for parsing, beyond which the Parser will write this “S” record into the unsupported directory. For more information about the unsupported direc-tory see section 8.10, “Ana-lyzing unsupported records”, on page 112.

--config-dir=CONFIGDIR where CONFIGDIR is the directory path to the location of the Parser configuration file. The default PBSA configura-tion directory is INSTALL_DIR_13.0/portal/ services/pbsa/parser/modules/pbsdata/config

Use this option to specify the location of the Parser config-uration file config.xml.

--show n/a. Use this option to display the current Parser configuration on the screen for review.





15.7.2 Examples

15.7.2.1 Configuring the Parser

This example shows how to configure the Parser to begin processing the PBS Professional accounting logs 30 minutes after the PBSA service is started, and to run the daily parsing cycle at 2:00 am.

pbsa-config-parser --parser-cycle-delay=30 --parser-timer="0 0 2 * * ?" --job-duration=60

15.7.2.2 Specify the location of the Parser configuration file

This example shows how to specify the location of the Parser configuration file con-fig.xml.

pbsa-config-parser --parser-cycle-delay=30 --parser-timer="0 0 2 * * ?" --config-dir=/tmp/pbsa-config-dir

15.7.3 Error Message



Troubleshooting

The PBSWORKS_COMMAND_HOME or PBSWORKS_SERVER_HOME variable was not set, and I was unable to find or open a pbsworks-pbsa.conf file.

The environment variables PBSWORKS_COM-MAND_HOME and PBSWORKS_SERVER_HOME have not been exported to the envi-ronment. Contact your systems administrator to set up the environ-ment properly.




The environment variables PBSWORKS_COM-MAND_HOME and PBSWORKS_SERVER_HOME have not been exported to the envi-ronment. Contact your systems administrator to set up the environ-ment properly.

Python version XX is not supported. Supported version is 2.4.x or higher.


Error occurred while get uid for user XXXX. Unable to get user id for current user. Contact your system admin-istrator to report this error.

Operation not permitted, Please run command as XXXX user or ROOT.


Option XXXX argument XXXX must be number. Try pbsa-config-parser --help

Then specified option must be expressed as a numeric.

Please specify at least one option either of --parser-cycle-delay=PARSER_CYCLE_DELAY, --parser-timer=PARSER_TIMER, or --job-duration=JOB_DURATION.

You must specify at least one of the following options:

--parser-cycle-delay

--parser-timer

--job-duration

The configuration file is empty. This error may occur after execut-ing this command with the --show option. The configuration file is empty, therefore nothing has been configured yet.



Troubleshooting



Config directory does not exists. The PBSA configuration directory INSTALL_DIR_13.0/por-tal/services/pbsa/parser/modules/pbsdata/config does not exist. An error may have occurred during installa-tion of PBSA. Please re-install the product.

Unable to instantiate the PBSA ToolKit. Please check your PBSWORKS_SERVER_HOME vari-able.

The environment variable PBSWORKS_SERVER_HOME may not have been exported to the environment. Contact your sys-tems administrator to set up the environment properly.

Unable to read the parser config file. Please check the existence and per-missions of the config.xml located at INSTALL_DIR_13.0/por-tal/services/pbsa/parser/modules/pbsdata/config



Troubleshooting



15.8 Resetting the PBSA dataset

Use the pbsa-data-reset command to:

• discard the data from the PBSA database

• delete the history files indicating which PBS Professional accounting log files have already been transported to the machine where the Parser is installed

• reset all Data Collectors


Stop the PBSA service before issuing this command. If this command is executed when the PBSA service and database are running, then this command will exit with an error.

15.8.1 Usage

pbsa-data-reset [options]

This command resets the PBSA database password back to “pbsworks@123”. You can use the pbsa-data-password command to change the default.



--version n/a Use this option to display the com-mand's version number. This option cannot be used in conjunc-tion with any other options.




15.8.2 Examples

15.8.2.1 Resetting a dataset

Executing this command discards the parsed data from the PBSA database and deletes the his-tory files that indicate which PBS Professional accounting log files were previously trans-ported.

pbsa-data-reset




Troubleshooting













Error in reading the inputs.

Please check log file: XXXX

An error occurred while retrieving system arguments. Contact your system adminis-trator to report this error.

Property with key XXXX doesn't exists. An error occurred while reading the data-base configuration file monetdb-pbsa.conf. Contact your system admin-istrator to report this error.

File XXXX must end with (.backup) An error occurred while resetting the data-set. Contact your system administrator to report this error.

File XXXX doesn't exists. An error occurred while reading the DBConnection.properties file. Contact your system administrator to report this error.

Error in reading monetdb-pbsa.conf file


An error occurred while reading the data-base configuration file monetdb-pbsa.conf. Contact your system admin-istrator to report this error.

Updation of config file datasources.xml failed.

An error occurred while updating the con-figuration file datasources.xml. Con-tact your system administrator to report this error.

Updation of config file .monetdb failed. An error occurred while updating the con-figuration file monetdb-pbsa.conf. Contact your system administrator to report this error.

Updation of config file monetdb-user.sql failed.

An error occurred while updating the file monetdb-user.sql. Contact your sys-tem administrator to report this error.



Troubleshooting



Error in splitting command: XXXX An error occurred while resetting the data-set. Contact your system administrator to report this error.

Error in executing command: XXXX An error occurred while resetting the data-set. Contact your system administrator to report this error.

Command splitting failed: XXXX An error occurred while resetting the data-set. Contact your system administrator to report this error.

Server startup failed. An error occurred upon database server start up. Contact your system administrator to report this error.

Server status setting failed. An error occurred while setting the database server status. Contact your system adminis-trator to report this error.

Server status check failed. An error occurred while checking the data-base server status. Contact your system administrator to report this error.

Datacollectors load history deletion failed. An error occurred while deleting the Data Collector load history file history.xml. Contact your system administrator to report this error.

Parser availability directory deletion failed.

An error occurred while deleting the Parser availability directory. Contact your system administrator to report this error.

PBSA parser cleaning failed. An error occurred while resetting the data-set. Contact your system administrator to report this error.



Troubleshooting



Data reset failed.


An error occurred while resetting the data-set. Contact your system administrator to report this error.


Unable to instantiate the PBSA ToolKit. Please check your PBSWORKS_SERVER_HOME variable.


Please make sure PBSA engine and Mon-etdb Server is not running.

Cannot reset the data as the PBSA service and database are running. See section 14.1.2, “Stopping the PBSA service”, on page 142 for more information.



Troubleshooting



15.9 Changing the PBSA database password

Use the pbsa-data-password command to change or reset the PBSA database pass-word.


The PBSA service must be stopped before issuing this command. If this command is exe-cuted when the PBSA service and database are running, then this command will exit with an error.

15.9.1 Usage

pbsa-data-password OLD_PASSWORD NEW_PASSWORD [options]

where:

OLD_PASSWORD is the current password

NEW_PASSWORD is the new password

The PBSA installer creates a database superuser with the following credentials:

user id: pbsworks

password: pbsworks@123



--version n/a Use this option to display the com-mand's version number. This option cannot be used in conjunc-tion with any other options.




15.9.2 Examples

15.9.2.1 Resetting the PBSA database password

Executing this command changes the current password “pwdJan2015” to a new password “pwdFeb2015”.

pbsa-data-password pwdJan2015 pwdFeb2015




Troubleshooting





Incorrect number of arguments. Try pbsa-data-password --help.




Error occurred while get uid for user xxxxxx.






Error in reading the inputs.


An error occurred while retrieving system arguments. Contact your system adminis-trator to report this error.

Property with key XXXX doesn't exists. An error occurred while reading the data-base configuration file monetdb-pbsa.conf. Contact your system admin-istrator to report this error.

File XXXX must end with (.backup) An error occurred while changing the pass-word. Contact your system administrator to report this error.

File XXXX doesn't exists. An error occurred while reading the DBConnection.properties file. Contact your system administrator to report this error.

Error in reading monetdb-pbsa.conf file


An error occurred while reading the data-base configuration file monetdb-pbsa.conf. Contact your system admin-istrator to report this error.

Error in splitting command: XXXX An error occurred while changing the pass-word. Contact your system administrator to report this error.

Error in executing command: XXXX An error occurred while changing the pass-word. Contact your system administrator to report this error.



Troubleshooting



Line must be (key=value). xxxxxxx There is a problem with the DBConnection.properties file. Contact your system administrator to report this error.

Updation of config file context.xml failed. An error occurred while updating the con-figuration file context.xml. Contact your system administrator to report this error.

Updation of config file .monetdb failed. An error occurred while updating the con-figuration file monetdb-pbsa.conf. Contact your system administrator to report this error.

Updation of config file monetdb-user.sql failed.

An error occurred while updating the file monetdb-user.sql. Contact your sys-tem administrator to report this error.

Command splitting failed: xxxxxx An error occurred while changing the pass-word. Contact your system administrator to report this error.

Server startup failed. An error occurred upon database server start up. Contact your system administrator to report this error.

Server status setting failed. An error occurred while setting the database server status. Contact your system adminis-trator to report this error.

Server status check failed. An error occurred while checking the data-base server status. Contact your system administrator to report this error.

Password change failed.


An error occurred while changing the pass-word. Contact your system administrator to report this error.



Troubleshooting



Please make sure PBSA engine and Mon-etdb Server is not running.

Cannot change the password as the PBSA service and database are running. See sec-tion 14.1.2, “Stopping the PBSA service”, on page 142 for more information.



Troubleshooting


Chapter 16

PBSA Measures & DimensionsPBSA supports a wide variety of fields or variables that can be analyzed and displayed via charts. These fields are divided into two categories:

• Dimensions are the independent variable (often denoted by the x-axis) whose variation does not depend on another variable. Examples are users, software, servers, and nodes. PBSA requires Dimensions be displayed on the x-axis or z-axis.

• Measures are the dependent variable (often denoted by the y-axis) whose value changes dependent on the other variables. Examples are jobs, memory, and cores. Measures are displayed on the y-axis. The PBSA Web Application refers to Measures as Values.

Two styles of chart reporting are available.

• Workload is the traditional way (PBS Analytics 10.x) of looking at the PBS Professional end record for each job. It does not take duration of the job into consideration. The fol-lowing resource usage is calculated for this type of reporting:

- Cores Requested

- Cores Used

- Memory Requested

- Memory Used


Chapter 16 PBSA Measures & Dimensions

• Utilization is used for reporting on resources for supply demand and billing use cases. This style of reporting takes into account the time over which the resource was consumed or demanded. The following resource usage is calculated for this type of reporting:

- Cores Queued

- Cores Used

- Memory Queued

- Memory Used

- Cores Queued Walltime

- Cores Used Walltime

- Memory Queued Walltime

- Memory Used Walltime

For more information about Reporting Styles see section 16.1.17, “Reporting Style”, on page 217.

Following is a list of Dimensions and Measures that are available with PBSA.

16.1 Dimensions

16.1.1 Account

Account is used as a grouping for charging for the use of resources. Account can also repre-sent usage of a group of users based on their respective accounts to present a bigger picture for charging for the use of resources.

This is extracted from the account attribute of the PBS Professional accounting log.

16.1.2 Architecture

This dimension represents the machine (hardware) architecture requested for running the job. This is used to report architecture wide usage across nodes within a cluster and to clarify what kind of architecture is most frequently requested. This information can be used to determine the cause of long job wait times due to hardware unavailability.

This is extracted from the Resource_List.arch attribute of the PBS Professional accounting log.


PBSA Measures & Dimensions Chapter 16

16.1.3 Core Count

This dimension represents the number of CPUs used by a job. This dimension can be used to filter a chart by the number of cores used, or to chart the number of cores used by a job.

This is extracted from the resources_used.ncpus attribute of the PBS Professional account-ing log.

16.1.4 Custom Group

Custom groups are used to track utilization at a project level for sites running versions of PBS Professional prior to 11.2, therefore a custom group is the project to which a user has been assigned. Custom groups are configured using the pbsa-config-groups command (see section 15.4, “Configuring custom groups”, on page 175).

Custom groups are used when the user has been assigned to a group other than their default group that is mentioned in the PBS accounting logs. Custom group charts can be generated so that managers can bill the project to which the user is currently assigned, rather than their default group.

This is a configurable value that is extracted from the Custom Group configuration file (see section 18.3, “Custom group configuration file”, on page 267).

16.1.5 Day Type

Day Type separates weekdays into the below categories. This information is configured when defining prime and non-prime time for the days of the week using the pbsa-config-hol-idays command (see section 15.3, “Configuring holidays”, on page 164). Use this dimen-sion to determine how many jobs ran during the weekend, or how much memory was used during a work day.

• Weekend - any day in the holiday configuration file where non-prime time is “ALL”

• Working Day - any day in the holiday configuration file where prime time is specified



16.1.6 Exit Reason

This dimension represents the way the job exited from the cluster, indicating whether it was successful or unsuccessful. It can be used to track job execution failure rates and reasons for failures with respect to various dimensions like jobs, nodes, memory, etc. Exit reasons are configured using the pbsa-config-exits command (see section 15.5, “Configuring exit codes”, on page 183).

This is a configurable value that is extracted from the Exit code configuration file (see section 18.2, “Exit code configuration file”, on page 264).

16.1.7 Exit Status

This dimension represents the exit status of the job, indicating the job was successful or unsuccessful. It is based on the predefined exit codes from PBS Professional. It can be used to track job execution failure rates based on the PBS Professional exit codes.

Exit Status is extracted from the Exit_status attribute of the PBS Professional accounting log.

16.1.8 Group

This dimension represents the Linux group (user group of the cluster) to which the user belongs. Using this dimension, a chart could be created that would show how a group is utiliz-ing the cluster.

This is extracted from the group attribute of the PBS Professional accounting log.

16.1.9 Job State

The state of the job. Charts can be generated to display resource usage by job state. Job state is determined based on information contained in the PBS Professional accounting log records (E-job ended, R-job reran, S-job started). Currently, the following job states are supported by PBSA:

• Finished - Job is finished

• Queued - Job is queued

• Running - Job is running



16.1.10 Node

This dimension represents the various execution nodes that belong to the cluster. Various charts can be generated to determine node utilization.

It is extracted from the exec_host attribute of the PBS Professional accounting log.

16.1.11 Node Class

This dimension represents a configurable classification or grouping of the execution MOM. This classification could be by hardware configuration or discipline. It is based on the needs of the site (see section 3.2.1, “Classifying the PBS Professional nodes”, on page 14). This dimension can be used to create a chart that would group similar type of nodes from the clus-ter for analysis of usage, downtime, etc.

This is extracted from the Node Details configuration file created during the configuration of PBSA (see section 18.5, “Node details file”, on page 275):

INSTALL_DIR_13.0/portal/services/pbsa/parser/modules/pbsdata/config/nodedetails_YYYY where "YYYY" is the four digit year.

16.1.12 Peering Direction

This dimension represents the peering direction. The value for this dimension will be “Fur-nishing Server -> Pulling Server”. In case there is no peering, this dimension will contain the server name.

16.1.13 Place

This dimension represents the placement of the job, indicating how it used the node resources - whether it was:

• exclusive

• free

• free:shared

• none

• pack

• scatter

It is extracted from the Resource_List.place attribute of the PBS Professional accounting log.



16.1.14 Prime - Non Prime

This dimension represents whether a job is being run during prime (normal working hours) or non-prime (non-working hours) hours. This is a configurable value that is extracted from the Holiday configuration file (see section 18.1, “Holiday configuration file”, on page 255).

Prime and non-prime time hours are configured using the pbsa-config-holidays com-mand (see section 15.3, “Configuring holidays”, on page 164).

16.1.15 Project

This dimension represents the project that the user is working on, it can be used to create proj-ect based charts for analysis of project resources such as users, nodes applications, etc.

This is extracted from the project attribute of the PBS Professional accounting log (see sec-tion 3.2.3, “Utilizing the qsub -P option”, on page 15).

16.1.16 Queue

This dimension defines the queue to which the job was submitted. This dimension could be used to create charts to determine the amount of jobs being submitted to a queue and for ana-lyzing queue wait times.

This is extracted from the queue attribute of the PBS Professional accounting log.



16.1.17 Reporting Style

This dimension allows you to select which style of calculations is implemented when deter-mining resource usage for consumable resources, such as cores and memory. There are two styles:

• Workload is the traditional way (PBS Analytics 10.x) of looking at the PBS Professional end record for each job. It does not take duration of the job into consideration. The fol-lowing resource usage is calculated for this type of reporting:

- Cores Requested

- Cores Used

- Memory Requested

- Memory Used

• Utilization is used for reporting on resources for supply demand and billing use cases. This style of reporting takes into account the time over which the resource was consumed or demanded. The following resource usage is calculated for this type of reporting:

- Cores Queued

- Cores Used

- Memory Queued

- Memory Used

- Cores Queued Walltime

- Cores Used Walltime

- Memory Queued Walltime

- Memory Used Walltime



The following tables compares and contrasts the resource calculations implemented by PBSA depending on the style of reporting:

Job identifier=1508.blrhpc3

queue time (qtime) = 4/15/2015 at 14:07:45 Resource_List.mem (kb) = 2048000

start time = 4/25/2015 at 14:08:15 resources_used.mem (kb) = 318548

end time = 4/19/2015 at 5:17:38 resources_used.ncpus = 8

Table 16-1: Job 1508.blrhpc3 Wait Time and Walltime per Day

4/15/2015 4/16/2015 4/17/2015 4/18/2015 4/19/2015

Wait time =

start time - qtime

30 secs 0 0 0 0

Walltime 35505 secs 86400 secs 86400 secs 86400 secs 19042 secs

Table 16-2: Workload Reporting

4/15/2015 4/16/2015 4/17/2015 4/18/2015 4/19/2015

Cores Requested 0 0 0 0 8

Cores Used 0 0 0 0 8

Memory Requested =

Resource_List.mem (kb)0

0 0 0 2048000

Memory Used =

resources_used.mem (kb)

0

0 0 0 318548


PB 219


4/18/2015 4/19/2015

C

W

0 * 8/86400 0 * 8/86400

C

W

86400 * 8/

86400

19042 * 8/

86400

M

W

0 * 318548/

86400

0 * 318548/

86400

M

W

86400 * 318548/86400

19042 * 318548/

86400

C

W

86400 * 8 19042 * 8

C

W

0 * 8 0 * 8

M

W

8 86400 * 318548

19042 * 318548

M

W

0 * 318548 0 * 318548


Table 16-3: Utilization Reporting

4/15/2015 4/16/2015 4/17/2015

ores Queued =

ait time * resources_used.ncpus/86400

30 * 8/

86400

0 * 8/86400 0 * 8/86400

ores Used =

alltime * resources_used.ncpus/86400

35505 * 8/

86400

86400 * 8/

86400

86400 * 8/86400

emory Queued =

ait time * resources_used.mem(kb)/86400

30 * 318548/86400

0 * 318548/86400

0 * 318548/86400

emory Used =

alltime * resources_used.mem(kb)/86400

35505 * 318548/86400

86400 * 318548/86400

86400 * 318548/

86400

ores Used Walltime =

alltime * resources_used.ncpus

35505 * 8 86400 * 8 86400 * 8

ores Queued Walltime =

ait time * resources_used.ncpus

30 * 8 0 * 8 0 * 8

emory Used Walltime =

alltime * resources_used.mem(kb)

35505 * 318548 86400 * 318548 86400 * 31854

emory Queued Walltime =

aittime * resources_used.mem(kb)

30 * 318548 0 * 318548 0 * 318548


16.1.18 Resource State

This dimension is only valid for consumable resources such as core, memory and node. It is the state of the resource:

• Available - Resources that are available across a PBS Complex. For instance, how many cores or how much memory is available across all nodes. This information is obtained when PBSA is configured using the output from pbsnodes -av.

• Down - Resources that are down. For instance, when a node is brought down then it’s resources such as CPU and memory are not available. This information is obtained when PBSA is configured using the output from pbsnodes -av.

• Queued - Resources that are explicitly requested at job submission or based on the poli-cies set via PBS Professional for the PBS Complex

• Unused - Resources that were not utilized - jobs did not run on that node therefore CPU and memory was not utilized.

• Used - Resources that were consumed.

16.1.19 Server

This dimension defines the server on which the job was submitted. This dimension can be used to create server wide charts in terms of resource usage.

This is extracted from the job identifier attribute of the PBS Professional accounting log.

16.1.20 Software

This dimension defines the software or solver for which the job has been submitted. This dimension can be used to create charts in terms of resource usage categorized by software.

This is extracted from the Resource_List.software attribute of the PBS Professional accounting log.

16.1.21 Time

This dimension allow resource usage to be reported by:

• Year

• Quarter

• Month

• Day



16.1.22 Time-Advanced

This dimension allow resource usage to be reported by:

Table 16-4: Time-Advanced Options

TimeOption

DescriptionExample

Last parsing cycle ran on August 29, 2015

Last Month Data from the previous month from the date of the most recent parsing cycle.

Chart will show data from July 1, 2015 to July 31, 2015.

Last Quarter Data from the previous quarter from the date of the most recent parsing cycle.

Chart will show data for 2015 Q2.

Last Week Data from the previous week (Sunday marks the beginning of a week) from the date of the most recent parsing cycle.

Chart will show data from August 17, 2015 to August 23, 2015.

Last Year Data from the previous year from the date of the most recent parsing cycle.

Chart will show data from January 1, 2014 to December 31, 2014.

This Month Data from the current month from the date of the most recent parsing cycle.


This Quarter Data from the current quarter from the date of the most recent parsing cycle.

Chart will show data for 2015 Q3.

This Week Data from the current week (Sunday marks the beginning of a week) from the date of the most recent parsing cycle.


This Year Data from the current year from the date of the most recent parsing cycle.

Chart will show data from January 1, 2015 to August 29, 2015.

Today Data from the current date of the most recent parsing cycle.

Chart will show data for August 29, 2015.

Yesterday Data from the previous day from the date of the most recent pars-ing cycle.

Chart will show data for August 28, 2015.



16.1.23 Time Holiday

This dimension represents a holiday that has been configured for the site during the installa-tion and configuration of PBSA. The pbsa-config-holidays command is used to con-figure holidays (see section 15.3, “Configuring holidays”, on page 164). This dimension can be used as a filter for determining which jobs ran during a holiday period.

This is a configurable value that is extracted from the Holiday configuration file (see section 18.1, “Holiday configuration file”, on page 255).

16.1.24 Time Week

This dimension allows resource usage to be reported by week. The beginning date of every week is considered to be Sunday.

16.1.25 User

This dimension represents the user who has submitted the job. This dimension can be used to create a resource utilization chart for a specific user.

This is the extracted from the user attribute of the PBS Professional accounting log.

16.1.26 Virtual Node

This dimension represents the various execution vnodes that belong to the cluster. This dimension could be used to create charts based on the vnodes in conjunction with other dimensions like Node Event Type that would give a picture of how the vnode was being uti-lized.

It is extracted from the exec_vnode attribute of the PBS Professional accounting log.



16.2 Measures

Following is a list of Measures that are available with PBSA:

Non-Consumable Resources

• Jobs - amount of jobs submitted to a PBS Professional cluster

• Overall Time - amount of time a job existed from its creation until the job exits.

• Unique Users - number of unique users that have submitted jobs on a cluster

• Wait Time - amount of time the job waited before it started running.

• Walltime - amount of time a job was in a running state, including the time it took to exit.

Consumable Resources

• Core (CPU) - amount of CPU

• Memory - amount of memory

• Node - number of nodes

16.2.1 Calculations for Non-Consumable Resources

Once a non-consumable resource is added to a chart, the chart builder may double-click on the resource and choose the aggregate function to apply to the resource.

Figure 16-1:Aggregate Functions

The aggregate functions available for non-consumable resources are:

• Sum - total of a resource, such as total number of jobs or total amount of wait time

• Avg - average of a resource, such as the average wait time (unavailable for the resources Jobs or Unique Users)



Therefore, the calculations available for non-consumable resources are:

The following sections in the PBSA 13.0 Chart Calculations document describe the non-con-sumable resource calculations:

• Calculations for Jobs

• Calculations for Overall Time

• Calculations for Unique Users

• Calculations for Wait Time

• Calculations for Walltime

16.2.2 Calculations for Consumable Resources

Reporting for consumable resources such as core, memory, and nodes can be done for the fol-lowing resource states:

• Available - amount of the resource that is available

• Unused - amount of the resource that is unused or not utilized

• Down - amount of the resource that is down or off-line

• Queued - amount of the resource that was queued (job was in a wait state)

• Used - amount of the resource that used by running jobs

Table 16-5: Available Calculations for Non-Consumable Resources

Resource SUM AVG

Jobs Total number of jobs N/A

Overall Time Total amount of overall time Average amount of overall time

Unique Users Total number of unique users N/A

Wait Time Total amount of wait time Average amount of wait time

Walltime Total amount of walltime Average amount of walltime



When reporting on consumable resources, it is recommended to either place the dimension Resource State on the z-axis, or filter the resource by a specific Resource State. This will provide more meaningful information. Otherwise, the reported total for the resource will be the summation of all resource states: Available + Unused + Down + Used + Queued.

Figure 16-2:Reporting on Consumable Resources

Two options are available for calculating consumable resources with PBSA. After adding a consumable resource to a chart, the chart builder may double-click on the resource and choose to enhance the calculations with walltime.

If “Enhance with Walltime” is enabled, resource walltime is calculated as implemented in pre-vious versions of PBS. If “Enhance with Walltime” is disabled, the calculations are based on the resource (how many cores, how much memory), rather than walltime.

Figure 16-3:Aggregate Functions



In addition, the chart builder can choose the aggregate function to apply to the selected resource. The aggregate functions available are:

• Sum - total of a resource, such as total number of cores or total amount of memory

• Avg - average of a resource, such as the average number of cores used across all jobs

• Percentage - percent of resource, such as the ratio of used cores vs. available cores

Therefore, the calculations available for consumable resources are:

The following sections in the PBSA 13.0 Chart Calculations document describe the consum-able resource calculations:

• Calculations for Cores

• Calculations for Memory

• Calculations for Nodes

Table 16-6: Available Calculations for Consumable Resources

Resource

Enhance w/WalltimeDisabled

Enhance w/WalltimeEnabled

SUM AVGPercenta

geSUM AVG

Percentage

Cores Total number of cores

Average number of cores

Percent of cores

Total amount of core walltime

Average amount of core walltime

Percent of core walltime

Memory Total amount of memory

Average amount of memory

Percent of memory

Total amount of memory walltime

Average amount of memory walltime

Percent of memory walltime

Nodes Total number of nodes

Average number of nodes

Percent of nodes

Total amount of node walltime

Average amount of node walltime

Percent of node walltime


Chapter 17

Troubleshooting

17.1 Log files and logging rollover policy

Use the following log files to troubleshoot problems with PBSA:

Data Collector Log File

PBSA Data Collector events and errors are recorded in this log file:

INSTALL_DIR_13.0/dc/logs/dc.log

Parser Log File

Events and errors that occur while parsing the PBS Professional accounting logs, and the exit status, custom groups, and custom resources configuration files are recorded in this log file:

INSTALL_DIR_13.0/portal/logs/parser.log

PBSWorks Log File

Events and errors that occur with the PBSA Web Application and associated services, as well as parsing of the node detail and holiday configuration files are recorded in this log file:

INSTALL_DIR_13.0/portal/logs/pbsworks.log

Apache Tomcat Server Log File

Apache Tomcat server events and errors are recorded in this log file:

INSTALL_DIR_13.0/portal/thirdparty/apache/tomcat/logs/cata-lina.out


Chapter 17 Troubleshooting

Log file rollover policy

A new log file is generated once the size of the current log file exceeds 100MB. The current log file will be renamed “logFileName.SeqNo” where SeqNo starts at 1 and increments by 1 every time the log file is rolled over. A maximum of 1000 backup files are kept.

17.2 Troubleshooting the installation of PBSA

17.2.1 Installer did not complete as expected

If the installer does not complete the installation as expected, please perform the steps below:

1. Remove the following file that is created by InstallAnywhere at the time of installa-tion:

/var/.com.zerog.registry.xml

2. Rerun the installer.

3. If the installer is still unable to complete normally, please contact the PBSA support team.

17.2.2 PBSA services user does not exist

I was installing PBSA and the following error message was displayed:

During installation, PBSA requests a userid under which all PBSA services will run. The use-rid that was specified does not exist on the server where PBSA is being installed. Re-enter a userid that is a pre-existing and preferably non-privileged user on the server where PBSA is being installed.

Specified username does not exist.

The specified user to register the PBSA services does not exist. Kindly enter a

different user name that exists on the machine.

Please choose "OK" to go back and enter a different user name or choose

"Cancel" to cancel and exit installation.


Troubleshooting Chapter 17

17.2.3 PBSA service user is blank

I was installing PBSA and the following error message was displayed:

During installation, PBSA requests a userid under which all PBSA services will run. The use-rid that was specified is blank. Re-enter a valid userid that is a pre-existing and preferably non-privileged user on the server where PBSA is being installed.

17.2.4 License server not provided at installation

I was installing PBSA and a message similar to the following warning was displayed:

It is not mandatory to provide the license server at installation time. Once PBSA is installed, you will need to configure the license server prior to starting the PBSA service (see section 19.3.1, “Initial license server configuration”, on page 306). If the PBSA service is started prior to configuring the license server, the Web Application will not be accessible due to a failure to check out the appropriate licenses (see section 17.4.1, “Failed to check-out license”, on page 233). Once the PBSA service is started, configuration of the license server requires a slightly different process - see section 19.3.2, “Configuring the license server after starting PBSA”, on page 308.

PBSA Service username cannot be blank.

The PBSA service user name cannot be blank. Kindly enter a valid user name that

exists on the machine for registering the PBSA services.

Please choose "OK" to go back and enter a valid user name or choose

"Cancel" to cancel and exit installation.

LIcense server information not provided

The license server information has not been supplied. You can continue by

clicking "OK" and provide the license server information later in the

/opt/altair/pbsa/13.0/portal/binaries/framework/data/factorySettings.xml file.

Please click "OK" to continue without providing the information OR click "Back"

to go back and provide the information OR click "Cancel" to exit installation



17.2.5 Tomcat failed to start

I was installing PBSA and the following error occurred when I tried to start the PBSA ser-vices:

Check the Apache Tomcat log file (INSTALL_DIR_13.0/portal/thirdparty/apache/tomcat/logs/catalina.out). If the following error message exists in the log file, then there is not enough memory for the Java Virtual Machine.

Please review the minimum hardware requirements necessary for PBSA, and make adjust-ments as necessary (see section 1.1, “System Requirements”, on page 2).

Otherwise, check all log files for error messages (see Chapter 16, "Log files", on page 211) and contact system support.

17.2.6 Determining the installation location of PBSA

Web application and Parser

The installation location of the Web Application and Parser is stored in the following file on the machine where the Web Application and Parser are installed:

/etc/pbsworks-pbsa.conf

Data collector

The installation location of the Data Collector is stored in the following file on the machine where the Data Collector is installed:

/etc/pbsworks-dc.conf

Tomcat Failed to start

----------------------

Tomcat installation might have failed.

Error occurred during initialization of VM

Could not reserve enough space for object heap

Could not create the Java virtual machine.



17.3 Troubleshooting the PBSA service startup

17.3.1 Tomcat failed to start

I was trying to start/restart PBSA and the following error occurred :

Check the Apache Tomcat log file (for more information about the PBSA log files see section 17.1, “Log files and logging rollover policy”, on page 227). If the following error message exists in the log file, then there is not enough memory for the Java Virtual Machine.

Please review the minimum hardware requirements necessary for PBSA, and make adjust-ments as necessary (see section 1.1, “System Requirements”, on page 2).


PBSWorks Login server startup failed. Please check your installation.

Error occurred during initialization of VM

Could not reserve enough space for object heap

Could not create the Java virtual machine.



17.3.2 PBSA unable to start due to a port conflict

I was unable to start/restart PBSA and the following error was found in the Apache Tomcat web server log file:

INSTALL_DIR_13.0/portal/thirdparty/apache/tomcat/logs/catalina.out

ERROR, 26 Sep 14 19:24:03,066 -

{Http11Protocol.java: 182}

- Error initializing endpoint

java.net.BindException: Address already in use: JVM_Bind <null>:9000

at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:549)

at org.apache.coyote.http11.Http11Protocol.init(Http11Protocol.java:180)

at org.apache.catalina.connector.Connector.initialize(Connector.java:1124)

at org.apache.catalina.core.StandardService.initialize(StandardService.java:703)

at org.apache.catalina.core.StandardServer.initialize(StandardServer.java:838)

at org.apache.catalina.startup.Catalina.load(Catalina.java:538)

at org.apache.catalina.startup.Catalina.load(Catalina.java:562)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:261)

at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)

Caused by: java.net.BindException: Address already in use: JVM_Bind

at java.net.PlainSocketImpl.socketBind(Native Method)

at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:383)

at java.net.ServerSocket.bind(ServerSocket.java:328)

at java.net.ServerSocket.<init>(ServerSocket.java:194)

at java.net.ServerSocket.<init>(ServerSocket.java:150)

at org.apache.tomcat.util.net.DefaultServerSocketFactory.createSocket(DefaultServerSocket-Factory.java:50)

at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:538)

... 12 more



This message indicates that there is a port conflict. A process is already using the port (see highlighted port in the above error message) which has been configured for PBSA. Two solu-tions are available for resolving the port conflict:

1. Use the Linux command netstat -tulpn or lsof -i to determine which pro-cess is using the port. Stop the process and then start PBSA.

2. Change the ports that are being used by PBSA - see section 19.12, “Changing the ports used by PBSA”, on page 319.

17.4 Troubleshooting logging into PBSA

17.4.1 Failed to check-out license

I was logging into PBSA and the following error occurred :

This message may be caused by two issues. First, the maximum number of licenses for PBSA is currently in use. The system administrator can query the license server for license usage to determine who has licenses checked out. Secondly, an invalid license server may have been provided during installation, or when the license server was configured post-installation. Please see section 19.3.2, “Configuring the license server after starting PBSA”, on page 308 to validate the license server and modify it if necessary.


17.4.2 Invalid userid or password

I was logging into PBSA and the following error occurred :

This message may be caused due to an invalid SSH server being provided during installation. Please see section 19.5, “Configuring SSH Authentication”, on page 312 to validate the SSH server and modify it if necessary.


Failed to check-out license

Incorrect Username or Password



17.5 Troubleshooting the Data Collector

17.5.1 Data collector is not collecting the accounting logs

I installed PBSA and started the Data Collectors. The Data Collector is not collecting the PBS Professional accounting logs. What is wrong?

Check the PBSA Data Collector log (see Chapter 16, "Log files", on page 211) for any error messages.

The following message indicates that the location of the PBS Professional accounting logs does not exist.

An invalid location may have been provided during installation of PBSA. Please see section 19.6, “Changing the accounting log location”, on page 313 to validate the location of the PBS Professional accounting logs and make modifications if necessary.

These messages indicate that the PBSA Data Collector was unable to establish communica-tion with the PBSA service.

The PBSA service must be started prior to starting the Data Collector. If the PBSA service has not been started, then:

• start the PBSA service

• restart the Data Collector services

If after starting the PBSA service this error persists, then an invalid port number may have been provided for the PBSA service during installation of the Data Collector. Please see sec-tion 19.7, “Configuring communication between the DC and the PBSA service”, on page 314 to verify the PBSA service port and make modifications if necessary.

DEBUG, 19 Mar 13 10:55:00,019 -{AccountingLogCollector.java: 51} - Specified Log Directory doesn't exist. path_to_acct_logs

ERROR, 25 Mar 13 09:34:29,471 -{DefaultCommService.java: 38} - register request encountered a problem.

Caused by: java.net.ConnectException: Connection refused

DEBUG, 25 Mar 13 09:34:29,473 -{DataCollectorController.java: 169} - Not able to register with server.



These messages indicate that the PBSA Data Collector was unable to establish communica-tion with the PBSA service due to an invalid hostname.

An invalid hostname may have been provided for the PBSA service during installation of the Data Collector. Please see section 19.7, “Configuring communication between the DC and the PBSA service”, on page 314 to verify the PBSA service hostname and modify it if neces-sary. Otherwise, check all log files for error messages (see Chapter 16, "Log files", on page 211) and contact system support.

These messages indicate that the PBSA service port (typically 9000) has not been opened for incoming TCP traffic so that the PBSA service and the Data Collector services can communi-cate. This error is seen in the Data Collector log file when the Data Collector attempts to transport the accounting logs to the machine where the Web Application and Parser are installed:

To correct this problem, open the PBSA service port for TCP traffic.

ERROR, 25 Mar 13 09:34:29,471 -{DefaultCommService.java: 38} - register request encountered a problem.

Caused by: java.net.UnknownHostException: invalidhostname

DEBUG, 25 Mar 13 09:34:29,473 -{DataCollectorController.java: 169} - Not able to register with server.

ERROR, 11 Sep 13 01:48:25,332 -{DataTransportManager.java: 91} - Not able to send file 20110503

com.altair.pbsworks.dc.framework.DCException: File Transport proxy encountered a problem while processign request.


at com.altair.pbsworks.dc.transport.DataTransportManager.transportFile(DataTrans-portManager.java:83)

at com.altair.pbsworks.dc.transport.DataTransportManager.doWork(DataTransportMan-ager.java:73)

at com.altair.pbsworks.dc.framework.AbstractWorker.run(AbstractWorker.java:79)


Caused by: java.io.IOException: Server returned HTTP response code: 500 for URL: http://192.168.33.244:9000/parser/upload

at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnec-tion.java:1436)

... 4 more



17.6 Troubleshooting licensing

17.6.1 Licensing record count has been exceeded

I get the following message when accessing chart information through the PBSA Web Appli-cation.

The PBS Professional accounting log license record count has exceeded the current licensing option. The PBSA Web Application will only display the most recent data based on the licensing option. To ensure access to all parsed data, upgrade the licensing option - see sec-tion 19.9, “Changing the license option”, on page 316.

17.7 Troubleshooting access to PBSA

17.7.1 PBSA is not accessible from a different machine

I have installed PBS Analytics and it is up and running. However, it is not accessible from a different machine. What could be wrong?

Check for following :

• Are all the ports used by PBSA open and accessible?

Or

• Stop the iptable by using below command:

/etc/init.d/iptables stop

Accounting log license record count has exceeded the installed licensing schema. Contact your PBSA system administrator to upgrade the licensing schema. Only data permitted by the active licensing schema will be displayed. The permitted time range is from YYYY-MM_DD

to YYYY-MM-DD.



17.8 PBSA Log File Messages

The following sections provide information about messages logged by PBSA. Logging mes-sage are grouped into the following categories:

• Informational - messages highlighting the progress of the application at a coarse-grained level.

• Warning- messages indicating a potential issue - action may be required.

• Error - events indicating a need for action.

17.8.1 Informational Messages

The following table is a list of informational messages that are logged by PBSA:

Table 17-1: Informational log messages

Message Description

**Starting PBS Analytics Parser component**** The service responsible for running the Parser is starting.

Analytics engine initialization is in progress, hence skipping this parse cycle.

Parsing and caching of the accounting logs did not complete before another Parser cycle initiated (it ran for more than 24 hours). The newest Parser cycle is skipped and those accounting logs that are not processed are picked up by the next day’s Parser cycle.

PBS accounting log parse cycle started <date-time>

Parsing of the PBS Professional accounting logs is starting on the date and time denoted by <datetime>.

Total Time Taken to parse <dc> accounting logs : <ss>

The time <ss>, in seconds, it took to parse the accounting logs transported from the Data Collector denoted by <dc>.

PBS accounting log parsing completed for <dc> Parsing of the accounting logs trans-ported from the Data Collector denoted by <dc> is completed.



Number of complete parse cycle : <xx> The number of parsing cycles denoted by <xx> that have completed.

Parser has completed parsing the accounting logs.

Parsing of the accounting logs is com-plete.

Analytics engine is now being initialized with the parsed accounting log data. Please do not login to the Web Application until this process is complete.

Data must be loaded into the PBSA database and cached before the Web Application is available.

To check for completion of the initialization pro-cess, view the Web Application log file pbsworks.log.

A message will be written to the pbsworks.log file when the Web Application is available.

Parse cycle delay : <time> The Parser cycle delay in minutes is set to the value of <time>. The pars-ing process will start after this delay expires.

Parser configuration location : <dir> The directory denoted by <dir> is where the PBSA configuration files (node details, holiday, exit status) are stored.

Configuration file for parser <filename> Location of the parser configuration file denoted by <filename>. This file contains information for configuring the parsing delay and when the daily Parser cycle is scheduled to run.

Server shut down called when parsing is in prog-ress

Parsing is in progress when the PBSA service is stopped or restarted. Pars-ing will continue when the PBSA ser-vice is started.

Number of records in recovery : <xx> The number of accounting log entries, denoted by <xx>, that had issues and were written to a file in the Recovery directory.


Message Description



Record will be written into recovery An issue was found with an account-ing log entry. The issue is logged and the entry is written to a file in the Recovery directory. A Recovery directory is created for each Data Col-lector. The Recovery directories are located in INSTALL_DIR_13.0/portal/services/pbsa/parser/modules/pbsdata. A systems administrator can edit these files and correct the issue with the accounting log entry. The corrected entries are parsed and loaded into the PBSA database during the next Parser cycle during RECOVERY MODE.

The parser is running in RECOVERY MODE The Parser is processing the account-ing log records that were written to the Recovery directory. Files are written to the Recovery directory when an issue is found with an accounting log entry. A Recovery directory is created for each Data Col-lector. The Recovery directories are located in INSTALL_DIR_13.0/portal/services/pbsa/parser/modules/pbsdata. A systems administrator can edit these files and correct the issue with the accounting log entry. The corrected entries are parsed and loaded into the PBSA database during the next Parser cycle during RECOVERY MODE.


Message Description



Going to delete corrected recovery file >> <file-name>

Accounting log entries in the Recov-ery directory are corrected, parsed, and loaded into the PBSA database, therefore the Recovery file denoted by <filename> is being deleted.

The parser is running in NORMAL MODE The Parser is running in NORMAL MODE meaning that newly trans-ported accounting logs are being parsed.

The parser is running in RUNNING MODE PBSA writes information about run-ning jobs to a directory during the parsing process. The files stored in the Running Job directory are pro-cessed during RUNNING MODE.

Rerun Jobs encountered multiple times : Duplicate records have been found in the PBS accounting log for a re-run job. The first record is processing and the second record is ignored. No action is required.

No files to parse in : <dir> No accounting logs were written to either the Recovery directory or to the Running Jobs directory.

Parsing <filename> log file The accounting log file denoted by <filename> is being parsed.

File is parsed: <filename> The accounting log file denoted by <filename> is parsed.

Deletion failed for <filename> Failed to delete an accounting log file that was parsed. No action is neces-sary. This file will be ignored in the next Parser cycle.


Message Description



Number of running jobs : <xx> This message is displayed once during a parsing cycle and shows how many running jobs, denoted by <xx>, were found during the parsing cycle.

Deleting Zombie Jobs PBSA is deleting Zombie Jobs - jobs that are missing an end record (“E”), and are missing resource information on the start record (“S”). PBSA ignores these jobs and continues pars-ing. No action required.

Zombie Jobs deletion complete PBSA is finished deleting Zombie Job - jobs that are missing an end record (“E”), and are missing resource information on the start record (“S”).

Parser Cache initialised successfully Caching is complete.

Importing Data for finished jobs Loading the PBSA database with fin-ished job information.

Importing Data for node data Loading the PBSA database with node detail information.

Data import finished for node data Loading of the PBSA database with node detail information is complete.

Importing Data Loading of data into the PBSA data-base is beginning.

Importing Data for running jobs Loading the PBSA database with run-ning job information.

Importing Data for running jobs completed Loading of the PBSA database with running job information is complete.

Importing Data for node data for running jobs Loading the PBSA database with node detail information for running jobs.


Message Description



No data available for Import No data is available to be loaded into the PBSA database.

Import of Data Complete Loading of PBSA database is com-plete.

Reading config file : <filename> Reading the PBSA configuration file denoted by <filename>.

Finished Reading file :<filename> Finished reading the PBSA configu-ration file denoted by <filename>.

*****No custom resource configured ***** There are no custom resources con-figured for this site.

Going to read custom resource configuration Beginning to process custom resources (PBSA Hooks).

You have provided a custom hook class for this resource, going to verify class name : <classname>

Verifying that the class denoted by <classname> that is associated with the custom resource exists.

Successfully able to load hook class Custom resource class is found and loaded.

Custom Resource processing finished Processing of custom resources is complete.

Time taken to execute Query is <ms> This message indicates that a query was executed (a chart was displayed via the Web Application) and it took <ms> milliseconds to execute the query.

Request received for starting parsing cycle. The Parser is sending a message to the PBSA Web Application indicating that parsing is starting. Users will be blocked from using the Web Applica-tion while parsing is in process.


Message Description



Request received for completing parsing cycle. The Parser is sending a message to the PBSA Web Application indicating that parsing is completed and caching is beginning. Users will be blocked from using the Web Application while caching is in process.

Query Cache initialised successfully. PBSA is caching the queries associ-ated with the PBSA default charts and user charts.

Unable to find config directory for holiday con-figurations.

The directory where the PBSA con-figuration files are located is miss-ing. Contact systems support.

Reading node details file to populate availabilty information

Reading the node detail configuration files.

OTB Cube caching started PBSA default chart are being cached.

OTB Cube caching completed in Caching of PBSA default charts is completed.

Initialization of the analytics engine completed successfully in <mm> minutes. You may now login to the PBSA web application using the URL http://<hostname>:<port>/pbsworks

Parsing and caching is complete. The Web Application is available.

Executing: <chart name> Caching the chart denoted by <chart name>

Total number of licensing records : <xx> Total number of licensing records denoted by <xx> found during the parser cycle. For more information about licensing records see section 3.3.4, “Licensing options”, on page 17,


Message Description



17.8.2 Warning Messages

The following table is a list of warning messages that are logged by PBSA:

Table 17-2: Warning log messages

Message Description

System memory check failed, Exit status: Exit-Status

PBSA was unable to check for the availability of memory for running PBSA. No action is necessary.

Problem while importing : <data> There was an issue while loading the PBSA database. Contact systems support.

<attr> and <datatype> are mandatory elements while configuring new custom resource, hence skipping this resource

Mandatory XML elements for defin-ing a custom resource are missing. This custom resource will not be pro-cessed. Add the missing XML ele-ments to the custom resource file, discard any parsed data, and reparse all PBS Professional accounting logs.

Record will be written into recovery An issue was found with an account-ing log entry. The issue is logged and the entry is written to a file in the Recovery directory. A Recovery directory is created for each Data Col-lector. The Recovery directories are located in INSTALL_DIR_13.0/portal/services/pbsa/parser/modules/pbsdata. Edit the Recovery file and correct the issue with the accounting log entry. The corrected entries are parsed and loaded into the PBSA database during the next Parser cycle during RECOV-ERY MODE.



Record will be written into unsupported folder PBSA found an accounting log entry that contains invalid data. The issue is logged and the entry is written to a file in the unsupported directory. An unsupported directory is cre-ated for each Data Collector. The unsupported directories are located in INSTALL_DIR_13.0/portal/services/pbsa/parser/modules/pbsdata. View the contents of a file in the unsupported directory to analyze the issue. You may need to make changes to the PBS Professional con-figuration to make the missing/invalid PBS attribute available in the accounting log record.

No value for attribute PBSA found an accounting log entry that contains invalid data. The issue is logged and the entry is written to a file in the unsupported directory.

config file not present A PBSA configuration file is missing. No action is necessary.

Not providing these node details configurations will result in product showing you limited set of reports. Please consult Administrator Guide for details of charts and features provided / affected by this configuration.

No action required. For more infor-mation on how the absence of a node detail information affects the PBSA default charts see Table 2-1, “Absence of node information and its affect on PBSA charts,” on page xii.


Message Description



Server Shutdown called, going out of parse directory.

Parsing was in progress when the PBSA service was stopped or restarted. The accounting log file that is currently being parsed will be com-pleted. Parsing will continue when the PBSA service is started. No action required.

Empty parse cycle for dc : <dc> No accounting logs were transported for the Data Collector denoted by <dc>. No action required.

Error while fetching timezone difference PBSA was unable to determine the timezone of the Data Collector. The accounting logs will be stamped with the timezone of the PBSA server. No action required.

Unable to create rc_event_resource_occupancy table.

Internal error. Contact systems sup-port.

Unable to create nodes table. Internal error. Contact systems sup-port.

As you crossed threshold of data as permitted by active licensing schema, sliding time window of time interval is in effect. Data will be available only for time range

The accounting log license record count has exceeded the installed licensing option. Only data permitted by the active licensing option will be displayed via the Web Application. For more information about upgrad-ing your licensing option see section 19.9, “Changing the license option”, on page 316.


Message Description




The following table is a list of error messages that are logged by PBSA:

Table 17-3: Error log messages

Message Description

Problem in invoking parser Unable to start the Parser. Contact systems support.

Exception while creating the recovery directory Unable to create a Recovery direc-tory. Contact systems support.

Exception while creating the recovery log file Unable to create a Recovery file Contact systems support.

Unable to parse the date format Internal error. Contact systems sup-port.

error while retrieving key performance logs PBSA logs performance information such as how long it takes to parse, cache, and the overall time for both activities. PBSA was unable to retrieve this information. No action is necessary.

Caching Status URL cannot be empty Contact systems support.

error while retrieving disk space Disk space issues. Contact systems support.

Eh-Cache configuration for parsercache doesn't exists


unable to initialise Terracotta Eh cache. Internal error. Contact systems sup-port.

Unable read table metadata while generating csv files


File couldn't be moved Internal error. Contact systems sup-port.



unable to write output file Internal error. Contact systems sup-port.

A wrong value is configured a To Date There is an issue with the format of a To Date in a node detail configuration file. PBSA will log additional infor-mation to help determine which node detail entry is invalid. Edit the node details file and correct the issue. The corrected entry will be picked up in the next Parser cycle.

Error while generating checksum for customre-sources.xml file.


Error while gernerating md5 checksum. Internal error. Contact systems sup-port.

Error while writting checksum file. Internal error. Contact systems sup-port.

Error while validation checksum for customre-sources.xml file.


Custom resource file updated, please do a data reset everytime customresources.xml file is updated.

A custom resource was added to the custom resource configuration file, but a data reset was not performed. Reset the PBSA data set and then restart PBSA. Once PBSA is started, all accounting logs will be reparsed. For more information about resetting data see Chapter 20, "Reparsing the accounting logs", on page 323.

error in loading custom hook class PBSA could not load the Java class associated with a custom resource. Contact systems support.


Message Description



Skipping configuration of hook for resource There was a problem with configur-ing a PBASA hook for the custom resource denoted by <resource>. Contact systems support.

Error while configuring custom resource There was a problem with configur-ing a custom resource hook. Contact systems support.

Unable read table metadata for rc_event_re-source_occupancy_base table


Error while calculating proportinate walltime Internal error. Contact systems sup-port.

Error while converting values Internal error. Contact systems sup-port.

Error while parsing for unknown reasons Internal error. Contact systems sup-port.

Error while parsing record, please see stack trace

An exception occurred while parsing a PBS accounting log record. Contact systems support.

Found multiple accounting events logged in a single accounting record, hence discarding it

Duplicate records have been found in the PBS accounting log for a re-run job. The first record is processing and the second record is ignored. No action is required.

Job array main job information is not supported, writing the following record to unsupported folder.

The main container accounting log record for a job array is ignored. PBSA will only process the subjobs of the job array. No action is required.

Could not determine user or group information reliably, hence discarding it.

The user or group under which the job ran is missing, therefore PBSA will discard the accounting log record. No action is required.


Message Description



Could not find job start time information in accounting record, hence discarding it.

The start time of the job is missing, therefore PBSA will discard the accounting log record. No action is required.

Could not find job qtime information in accounting record, hence discarding it.

The time the job was queued is miss-ing, therefore PBSA will discard the accounting log record. No action is required.

Could not find job end time information in accounting record, hence discarding it.

The time the job ended is missing, therefore PBSA will discard the accounting log record. No action is required.

Could not find job used memory information in accounting record, hence discarding it.

The memory used by the job is miss-ing, therefore PBSA will discard the accounting log record. No action is required.

Could not find job execution hosts information in accounting record, hence discarding it.

Execution host information is miss-ing, therefore PBSA will discard the accounting log record. No action is required.

Could not connect to the database Internal error. Contact systems sup-port.

Error while checking configuration file existence Either the exit status or custom group configuration files are missing. Con-tact systems support.

Incorrect Date Format Encountered: Internal error. Contact systems sup-port.

Monetdb database backup failed. Internal error. Contact systems sup-port.

Exception while converting the memory attri-butes to KB



Message Description



error occurred while populating configurations There is an issue with either the exit status or custom group configuration file. Review any changes you may have made to these files and make modifications as necessary.

Please set the VM parameter for the configura-tion file


Error while reading ThreadPool Scale Factor from parser configuration.


Caching Status URL cannot be empty. Internal error. Contact systems sup-port.

Unable to send request for caching status. Verify that the PBSA Service is run-ning. If it is, then contact systems support.

Complete parse cycle URL cannot be empty. Internal error. Contact systems sup-port.

Unable to send request for complete parse cycle. Internal error. Contact systems sup-port.

Start Parse Cycle URL cannot be empty Internal error. Contact systems sup-port.

Unable to send request for start parse cycle. Internal error. Contact systems sup-port.

Error writing load history file Internal error. Contact systems sup-port.

Please check the accounting log file name, it should be of the format 'yyyymmdd'

A file was found while parsing the PBS Professional accounting logs that does not adhere to the naming con-vention “YYYYMMDD”. PBSA will ignore this file and continue parsing. No action required.


Message Description



Future date accounting log file A file was found while parsing the PBS Professional accounting logs that adheres to the naming convention “YYYYMMDD”, however the date is a future date. PBSA will ignore this file and continue parsing. No action required.

Error while creating start record Internal error. Contact systems sup-port.

Error while fetch start value for S jobs Internal error. Contact systems sup-port.

Error occured while executing report Internal error. Contact systems sup-port.

Not able to load cache. Internal error. Contact systems sup-port.

Eh-Cache configuration for query cache doesn't exists.


Error in holiday file in line <line> There is a problem with the holiday confiuration file in the line denoted by <line>. Edit the holiday file and correct the issue. The corrected entry will be picked up in the next Parser cycle.

Error while polulating Holiday cache. Internal error. Contact systems sup-port.

Error while executing Post Parse cycle activities Internal error. Contact systems sup-port.

Unable to complete Cache Priming. Internal error. Contact systems sup-port.

OTB Cache Priming failed. Internal error. Contact systems sup-port.


Message Description



Error Occured for User: <user> Internal error. Contact systems sup-port.

Problem occured while caching Internal error. Contact systems sup-port.

Error reading Chart: Internal error. Contact systems sup-port.

Exception while creating the directory Internal error. Contact systems sup-port.

Not able to load pbsaconfig.properties. Internal error. Contact systems sup-port.

Unable to retrieve licencs dates. Internal error. Contact systems sup-port.

Unable to read license information from file Internal error. Contact systems sup-port.

Exception Thrown ,could not connect to the database


Error while executing : <query> Internal error. Contact systems sup-port.

ERROR: NumberFormatException while pars-ing the PROCESS_MONITROING_INTER-VAL :



Message Description


Chapter 18

PBSA configuration filesConfiguration information for PBSA is stored in a set of files. These files are located on the machine where the PBSA Web Application and Parser are deployed. Configuration files are located in the following directory:


PBSA provides default configuration files for holidays, prime/non-prime time hours, and cus-tom groups for the last five years. It also supplies a default configuration file for exit codes. Do not delete these files.

Comments (lines denoted by a “#” at the beginning of the line) in configuration files are man-datory and should not be removed.

18.1 Holiday configuration file

The holiday schedule and prime (normal working hours) and non-prime (non-working hours) hours, are maintained by year in the file:

holiday_YYYY.conf

where "YYYY" is the four digit year.

The pbsa-config-holidays command (see section 15.3, “Configuring holidays”, on page 164) can be used to add information to this file.

By default, PBSA is configured with a single holiday, New Year’s day, for the years 2011 through 2015.


Chapter 18 PBSA configuration files

18.1.1 Defining a holiday

The following information is necessary to define a holiday and must be separated by a tab character. The holiday information must be stored below the prime and non-prime time infor-mation:

• The date of the holiday in YYYY-MM-DD format

• Holiday identifier or the name of the holiday

• Number of days the holiday will be celebrated

18.1.1.1 Holiday layout

Table 18-1: Holiday record layout

Column Name Description Data Type Valid ValuesMandatory/Optional

date(yyyy-mm-dd) The beginning date of the holiday

Date

YYYY-MM-DD

Date must be within range of January 1st and December 31st of the year for which you are defining holidays

Mandatory

holiday_type Holiday identifier or the name of the holiday

String Only restriction is the use of the backslash (\) character.

Mandatory

number_of_days Number of days the holiday will be celebrated

Integer Must be greater than or equal to 1

Mandatory


PBSA configuration files Chapter 18

18.1.1.2 Default holiday configuration data

Default holiday configuration file for 2011:





### CONFIGURE HOLIDAYS ###

#DATE(yyyy-mm-dd) HOLIDAY_TYPE NUMBER_OF_DAYS2011-01-01 New Years Day 1











18.1.1.3 Sample holiday data

Here is a sample of holiday data for the year 2015 stored in the file holiday_2015.conf.


#DATE(yyyy-mm-dd) HOLIDAY_TYPE NUMBER_OF_DAYS2015-01-01 New Years Day 12015-01-19 Martin Luther King Day 12015-04-03 Good Friday 12015-05-25 Memorial Day 12015-07-04 Independence Day 12015-09-07 Labor Day 12015-11-26 Thanksgiving 22015-12-25 Christmas Day 42015-12-31 New Years Eve 1



18.1.2 Defining prime and non-prime hours

The following information is necessary to define prime time (normal working hours) and non-primetime (non-working) hours to PBSA. The data must be separated by a tab character and must be stored above the holiday information:

• The day of the week

• Start time of the primetime hours

• Start time of the non-primetime hours

By default, PBSA is configured with prime and non-prime time hours for the years 2011 through 2015. Prime times hours for Monday through Friday begin at 09:00 am. Non-prime time hours begin at 5:30 pm. For Saturday and Sunday, the entire day is scheduled as non-prime time.



2 S Analytics 13.0 Administrator’s Guide

1

t

Mandatory/Optional

rs for weekdays (Mon-n specified separately.

nday - Friday has d.

Mandatory

rs for Monday

rs for Tuesday

rs for Wednesday

rs for Thursday

rs for Friday

rs for Saturday

rs for Sunday

rs for Saturday &

60 PB

8.1.2.1 Prime and non-prime time hours record layout

Table 18-2: Prime/non-prime time record layou

Column NameColumn

DescriptionData Type Valid Values

day day of week String weekdays prime/non-prime houFri) that have not bee

Cannot be used if Moalready been specifie

monday prime/non-prime hou

tuesday prime/non-prime hou

wednesday prime/non-prime hou

thursday prime/non-prime hou

friday prime/non-prime hou

saturday prime/non-prime hou

sunday prime/non-prime hou

weekends prime/non-prime houSunday

PB 261


p rs start Mandatory

red non-primetime for

should be set to "all”.

red primetime for the

should be set to

n hours start Mandatory

red primetime for the

ld be set to"all".

red non-primetime for

ld be set to "none".

t

Mandatory/Optional


rimetime start Start time of prime hours

Time

format=MM:SS

MM:SS Time prime time hou

none All hours are considethe day.

Non-primetime hours

all All hours are consideday.

Non-primetime hours"none".

on-primetime start Start time of non-prime hours

Time

format=MM:SS

MM:SS Time non-prime time

none All hours are consideday.

Primetime hours shou

all All hours are considethe day.

Primetime hours shou

Table 18-2: Prime/non-prime time record layou

Column NameColumn



18.1.2.2 Default prime and non-prime configuration data

Default prime/non-prime time configuration file for 2011:

Default prime/non-prime time configuration file for 2012: .



### CONFIGURE PRIME TIME AND NON PRIME TIME ###

#DAY PRIMETIME_START NON-PRIMETIME_STARTweekdays 09:00 17:30saturday none allsunday none all










18.1.2.3 Sample prime and non-prime hours data

Here is a sample of prime and non-prime time hours for the year 2015 stored in the file hol-iday_2015.conf.




#DAY PRIMETIME_START NON-PRIMETIME_STARTmonday 08:30 17:00weekdays 09:00 17:30saturday all nonesunday none all



18.2 Exit code configuration file

In order to track job execution failure rates and reasons for failures, it is recommended that PBSA be configured with the exit codes that are used at your site. PBSA is configured by default with two exit reasons “Successful” and “Unsuccessful” that are mapped to specific exit codes. An example of this default mapping can be found in section 18.2.3, “Default exit code configuration data”, on page 266. The following file stores this mapping:

exit_status.conf

The pbsa-config-exits command (see section 15.5, “Configuring exit codes”, on page 183) can be used to add information to this file.

18.2.1 Defining exit codes

The following information is necessary to map reasons to exit codes and must be separated by a tab character:

• A description that is mapped to exit codes

• Application or solver name which allows exit codes to be categorized by solver

• Exit codes or range of exit codes

The default exit code,“Still-Running” must not be removed from the exit code configuration file


PB 265


18

Mandatory/Optional

R "User delete" Mandatory

map any exit codes ed in the accounting re not directly this file to a reason.

A ll applications Mandatory

PBSJobApp

E 271 Mandatory

1-4

1,5,500


.2.2 Exit code layout

Table 18-3: Exit code record layout

Column NameColumn


EASON A description that will be mapped to exit codes

String A string of characters specified within double quotes. Single quotes (') are not allowed.

Example:

Unspecified Use this toencounterlogs that amapped in

PPLICATION Categorizes the exit codes by application or solver

String All Denotes a

Application Name Example:

XIT_STATUS Exit codes to be mapped to the reason/application

String A single exit code Example:

Range of exit codes Example:

List of comma separated exit codes

Example:



1

D

7,-18,-19,-30,31,32,33,34,35,36,37,38,39,40,68,69,70,71,72,73,74,75,76,77,7104,105,106,107,108,109,110,111,32,133,134,135,136,137,138,139,10,161,162,163,164,165,166,167,16,189,190,191,192,193,194,195,196217,218,219,220,221,222,223,224,45,246,247,248,249,250,251,252,2

66 PB

8.2.3 Default exit code configuration data

efault exit code configuration file: .

#REASON APPLICATION EXIT_STATUS# Private Setting"Still Running" All Still-Running# End Private Setting"Successful" All 0"Unsuccessful" All -1,-2,-3,-4,-5,-6,-7,-8,-9,-10,-11,-12,-13,-14,-15,-16,-120,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,678,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,253,254,255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271


18.2.4 Sample exit code data

Here is a sample of a possible exit code mapping stored in the exit_status.conf file:

18.3 Custom group configuration file

Users can be assigned to specific custom groups (i.e., projects) by updating the file:

custom_group_YYYY.conf

where "YYYY" is the four digit year.

The pbsa-config-groups command (see section 15.4, “Configuring custom groups”, on page 175) can be used to add information to this file.

18.3.1 Defining custom group assignments

The following information is necessary to assign a user to a custom group. The data must be separated by a tab character:

• User login id

• Custom group (project) the user is assigned to

• Date on which the user began working on the project

• Date on which the user stopped working on the project

• User’s default custom group - this is used when the user is assigned to multiple projects

#REASON APPLICATION EXIT_STATUS# Private Setting"Still Running" All Still-Running# End Private Setting"User delete" All 271"Successful" All 0"PBSJobApp unsuccessful" PBSJobApp 1,5,10027"Unsuccessful" All 1-4,303,500"Unspecified" All 16024

The custom group “OTHERS” should NOT be removed from the custom group configuration file.




1

luesMandatory/

Optional

ple: asmith Mandatory

ple: WINDTUNNEL Mandatory

January 1st and which you are assigning

Mandatory

ust be within range of ry 1st and December 31st year for which you are ing users to groups

ust be greater than or to the FROM_DATE

Mandatory

fies that user is currently ng in this group

s default project - used a user is assigned to le projects. Utilization

e tracked against the lt project if the project is signated when the job is

itted (qsub -P option).

Optional

68 PB

8.3.2 Custom group record layout

Table 18-4: Custom group record layout

Column NameColumn

DescriptionData Type Valid Va

USER User login id. String Alphanumeric Exam

CUSTOM_GROUP Project that user is assigned to.

String Alphanumeric Exam

FROM_DATE Date user began working on project

Date

format=YYYY-MM-DD

Date must be within range ofDecember 31st of the year forusers to groups.

TO_DATE Assignment complete date.

Date

format=YYYY-MM-DD

YYYY-MM-DD Date mJanuaof theassign

Date mequal

now Speciworki

DEFAULT User’s default project - used when a user is assigned to multiple work projects.

Boolean true User’whenmultipwill bdefaunot desubm

PB 269


18

De

De

De

#####s

#####s

#####s


.3.3 Default custom group configuration data

fault custom group configuration file for 2011:

fault custom group configuration file for 2012: .

fault custom group configuration file for 2013: .

PBSA Custom Group configuration file

History:USER CUSTOM_GROUP FROM_DATE TO_DATE DEFAULTystem OTHERS 2011-01-01 2011-12-31






Default custom group configuration file for 2014: .

Default custom group configuration file for 2015: .

# PBSA Custom Group configuration file###History:#USER CUSTOM_GROUP FROM_DATE TO_DATE DEFAULTsystem OTHERS 2014-01-01 2014-12-31

# PBSA Custom Group configuration file###History:#USER CUSTOM_GROUP FROM_DATE TO_DATE DEFAULTsystem OTHERS 2015-01-01 2015-12-31



18.3.4 Sample custom group data

Here is a sample of possible custom group definitions stored in the custom_group_2015.conf file:

## PBSA Custom Group configuration file###History:#USER CUSTOM_GROUP FROM_DATE TO_DATE DEFAULT# Private Settingsystem OTHERS 2015-01-01 2015-12-31#End of Private Settingsystem OTHERS 2015-01-01 nowasmith PDD1 2015-01-01 2015-06-01 trueasmith SYS1 2015-01-01 2015-09-01 bnelson SYS1 2015-01-01 nowfrichards QA 2015-01-01 nowgcorning PDD1 2015-01-01 nowbnelson QA 2015-01-01 nowdbarrett PDD1 2015-01-01 2015-05-31dbarrett SYS1 2015-06-01 2015-12-31escott PDD1 2015-01-01 2015-03-31escott PDD1 2015-09-01 2015-12-31



18.3.5 Situations that require editing the Custom Group Configuration file

There are several situations that require editing the Custom Group configuration file:

• When a user is assigned to the same project more than once in a year.

For example, the user “escott” is assigned to the project “PDD1” from January - March 2015, and then again from September - December 2015.

• Defining a default group for a user.

A default group is designated for a user so that PBSA can track utilization when a user is assigned to multiple projects during the same time period, and a job is submitted without designating the project (using the qsub -P option). To set a default group for a user the custom group configuration file must be manually edited.

For example, “asmith” is assigned to the projects “PDD1” and “SYS1” during the same time period as designated below. “PDD1” is designated as the “default” project with the addition of “true” in the DEFAULT column. If “asmith” submits a job without designat-ing the project, then PBSA will track the job’s utilization against “PDD1”, since it is des-ignated as the “default” project.

#USER CUSTOM_GROUP FROM_DATE TO_DATE DEFAULTescott PDD1 2015-01-01 2015-03-31escott PDD1 2015-09-01 2015-12-31

#USER CUSTOM_GROUP FROM_DATE TO_DATE DEFAULTasmith PDD1 2015-01-01 2015-06-01 trueasmith SYS1 2015-01-01 2015-06-01



18.3.6 Tracking utilization using custom groups

When a user submits a job, PBSA will use the custom group mapping file to determine the group under which the utilization will be tracked.

• User "gcorning" submits a job.

PBSA will track the usage against the group "PDD1" because "gcorning" has only been assigned to that group.

• User "dbarrett" submits a job in March 2015.

PBSA will track the usage against the group "PDD1", because "dbarrett" worked for that group from January through May of 2015.

• User "asmith" submits a job in April 2015.

PBSA will track the usage against the group "PDD1", even though "asmith" worked for both the "PDD1" and "SYS1" groups at that time. PBSA will attribute the usage to that group because it has been denoted as the default.

• User "asmith" submits a job in September 2015.

PBSA will track the usage under the group "SYS1", because "asmith" was only assigned to that group at that time.:

# PBSA Custom Group configuration file##USER CUSTOM_GROUP FROM_DATE TO_DATE DEFAULT# Private Settingsystem OTHERS 2015-01-01 2015-12-31#End of Private Settingsystem OTHERS 2015-01-01 nowasmith PDD1 2015-01-01 2015-06-01 trueasmith SYS1 2015-01-01 2015-09-01 bnelson SYS1 2015-01-01 nowfrichards QA 2015-01-01 nowgcorning PDD1 2015-01-01 nowbnelson QA 2015-01-01 nowdbarrett PDD1 2015-01-01 2015-05-31dbarrett SYS1 2015-06-01 2015-12-31escott PDD1 2015-01-01 2015-03-31escott PDD1 2015-09-01 2015-12-31



18.4 Custom resource configuration file

PBSA reports on various common PBS Professional resources such as CPUs, memory, and walltime. In addition, PBSA can report upon site-specific resources known as custom resources. This type of reporting requires a PBSA Hook, a mechanism for adding a custom resource to PBSA. Once a custom resource is configured, it can be added to a chart either as a Dimension or a Measure. For more information about creating a PBSA Hook and the custom resource configuration file see Chapter 23, "PBSA Hooks", on page 335.



18.5 Node details file

Information about each PBS complex within a cluster improves PBSA’s ability to provide accurate HPC utilization reporting. A description or map of a PBS complex includes informa-tion about each PBS Professional server and its associated execution hosts.

PBSA stores the description of each PBS complex in a series of files called the node details files. Essentially, the information in these files is a map of your PBS Professional cluster. These maps are stored by year in the file nodedetails_YYYY.conf where "YYYY" is the four digit year. These files are stored in the following location:


Use the pbsa-config-nodes command to create the node details files. This command builds the node details files using information obtained by running the PBS Professional com-mand pbsnodes -av. For more information see Chapter 9, "Supplying PBS node infor-mation", on page 113.

Once these node details files are created, the current year’s node details file will manually have to be edited when changes occur to the PBS cluster. Changes can include the following:

• adding a new node

• deleting a node

• taking a node offline

• bring a node back online

These four events are described by appending a new set of records to the current year’s node details file. This information will be processed in the next Parser cycle. For more informa-tion about modifying the current year’s node details file see the following sections:

• section 18.5.8, “Adding a node example”, on page 297

• section 18.5.9, “Bring a node down example”, on page 298

• section 18.5.10, “Deleting a node example”, on page 299



18.5.1 Organization of node detail file

Describing a PBS complex requires multiple rows of data and each row has its own record type and corresponding format. The following record types are used to describe your site’s PBS Professional complex:

1. Event - indicates whether a server or node is being taken offline, deleted or added

2. Server - describes a PBS Professional server

3. Node - describes a PBS Professional execution host

4. Vnode - describes a PBS Professional virtual node

5. Resource - describes the resources available on an execution host or a virtual node

Here is an example of how the data is organized in this file:

The following sections describe each record type of the PBSA node details configuration file.

1. Defining an event

2. Defining a PBS Professional server

3. Defining an execution host

4. Defining a virtual node

5. Defining resources

EVENT 1 SERVER A NODE A1 RESOURCE 1 RESOURCE 2 NODE A2 RESOURCE 1 SERVER B NODE B1 RESOURCE 3 NODE B3 RESOURCE 4EVENT 2 SERVER C NODE C1 VNODE 1 RESOURCE 1 VNODE 2 RESOURCE 1



18.5.2 Defining an event

The first record type that is necessary to describe a PBS Professional complex to PBSA is the EVENT record. The EVENT record describes the action that is taking place, for instance tak-ing an execution host offline. The following information is necessary to define an EVENT record and must be separated by a tab character:

• EVENT_TYPE - Must be placed in the 1st column

The EVENT_TYPE can be one of the following:

• ADD - add a PBS Professional server or execution host

• DELETE - delete a PBS Professional server or execution host

• DOWN - take a PBS Professional server or execution host offline

• INSTALL_SNAPSHOT - describe to PBSA your site’s current PBS Professional complex configuration

• SERVER_NAME - Must be placed in the 3rd column

18.5.2.1 EVENT record layout

Table 18-5: EVENT record layout

Column Name

ColumnNumber

DescriptionData Type

Valid Values

Mandatory

EVENT_TYPE 1 Defines the type of event such as adding a node or taking a node offline

String ADD Yes

DELETE

DOWN

INSTALL_SNAPSHOT

SERVER_NAME 3 Server name as defined in PBS Professional

String Must be a valid PBS Professional server.

Yes




1

T SNAPSHOT event is used to describe th BS Professional server called P lex after installation of PBSA, but prior to

T ex. The node “blrhpc3n30” is being d

CE_DATATYPE RESOURCE_TYPE BUILT-IN FURNISHING_SERVERS - - - - - 2015-01-01 now - - - 2015-01-01 now - - -ux - - - custom -gnn - - - custom -84kb - - - custom - - - - custom -gnn - - - custom -RE1GB - - - custom -

CE_DATATYPE RESOURCE_TYPE BUILT-IN FURNISHING_SERVERS - - -- 2015-11-01 now - - - - - - custom -b - - - custom - - - - custom -4GB - - - custom -

78 PB

8.5.2.2 Sample EVENT record

he highlighted record below is an example of an EVENT record. In this example, the INSTALL_e configuration of the current year’s PBS Professional complex. The PBS complex has a single PBSServer. The INSTALL_SNAPSHOT event is used to describe the configuration of a PBS comp starting the PBSA service and processing the PBS Professional accounting logs.

he following example of an EVENT record is used to delete or remove a node from a PBS compleleted on November 1st, 2015.

#EVENT_TYPE ARG SERVER_NAME FQDN NODE_NAME VNODE VNODE_RANGE RESOURRESOURCE_NAME RESOURCE_VALUE FROM_DATE TO_DATE COMMENT CUSTOM_OR_INSTALL_SNAPSHOT RANGE PBSServer - - - - - - - -- - PBSServer PBSServer.prog.com - - - - - - -- - PBSServer - ux-gnn - - - - - -- - - - - - - string STATIC resources_available.arch lin- - - - - - - string STATIC resources_available.host ux-- - - - - - - integer STATIC resources_available.mem 163- - - - - - - integer STATIC resources_available.ncpus 2 - - - - - - - string STATIC resources_available.vnode ux-- - - - - - - string STATIC resources_available.nodeclass 2CO

#EVENT_TYPE ARG SERVER_NAME FQDN NODE_NAME VNODE VNODE_RANGE RESOURRESOURCE_NAME RESOURCE_VALUE FROM_DATE TO_DATE COMMENT CUSTOM_OR_DELETE RANGE - - - - - - - - - - - - - blrhpc3 blrhpc3.prog.com blrhpc3n30 - - - - - - - - - - - - string STATIC resources_available.arch linux- - - - - - - integer STATIC resources_available.mem 16384k- - - - - - - integer STATIC resources_available.ncpus 2 - - - - - - - string STATIC resources_available.nodeclass 1CORE


18.5.3 Defining a PBS Professional server

The second record type that is necessary to describe a PBS Professional complex to PBSA is the SERVER record. The SERVER record describes the PBS Professional server. The fol-lowing information is necessary to define a SERVER record and must be separated by a tab character:


• FQDN - Must be placed in the 4th column

• FROM_DATE - Must be placed in 12th column

• TO_DATE - Must be placed in 13th column

• COMMENT - Must be placed in the 14th column

• FURNISHING_SERVERS - Must be placed in 16th column




1

alid alues

Mandatory

fessional server. Yes

t of the pbsa-setup-hostname -f command.

Yes

Date must be less than the TO_DATE

Yes

Date must be greater than the FROM_DATE

Yes

Designates that the event is currently in effect

WN

No

fined in PBS Professional. No

80 PB

8.5.3.1 SERVER record layout

Table 18-6: SERVER record layout

Column Name

ColumnNumber

Column Description

Data Type

VV

SERVER_NAME 3 Server name as defined in PBS Professional

String Must be a valid PBS Pro

FQDN 4 Fully qualified domain name (FQDN) of the machine where the Data Collector is installed

String Obtained from the outpucounsellor script, or the

FROM_DATE 12 Date the event started Date YYYY-MM-DD

TO_DATE 13 Date the event ended Date YYYY-MM-DD

String now

COMMENT 14 Descriptive reason for the event

String Ex:

SCHEDULE_SHUTDO

HARDWARE_ISSUE

NETWORK_FAILURE

FURNISHING_SERVERS 16 If peering is enabled, then this is a comma separate list of the Furnishing servers.

String Furnishing servers as de

PB 281


18

Th NAPSHOT event is used to describe the co Professional server called PBSServer. Th rrently is running).

Th ve been added to the record.

# CE_DATATYPE RESOURCE_TYPE R BUILT-IN FURNISHING_SERVERSI - - - - -- 2015-01-01 now - - -- 2015-01-01 now - - -- ux - - - custom -- gnn - - - custom -- 84kb - - - custom -- - - - custom -- gnn - - - custom -- RE1GB - - - custom -

# CE_DATATYPE RESOURCE_TYPE R BUILT-IN FURNISHING_SERVERSI - - - - -- 2015-01-01 now - - Troy,MV- 2015-01-01 now - - -- ux - - - custom -- gnn - - - custom -- 84kb - - - custom -- - - - custom -- gnn - - - custom -- RE1GB - - - custom -


.5.3.2 Sample SERVER record

e highlighted record below is an example of a SERVER record. In this example, the INSTALL_Snfiguration of the current year’s PBS Professional complex. The PBS complex has a single PBSe date range indicates that the server has been up and running from January 1, 2015 to “now” (cu

e following examples describes a PBS complex where peering is enabled. Furnishing servers ha

EVENT_TYPE ARG SERVER_NAME FQDN NODE_NAME VNODE VNODE_RANGE RESOURESOURCE_NAME RESOURCE_VALUE FROM_DATE TO_DATE COMMENT CUSTOM_OR_NSTALL_SNAPSHOT RANGE PBSServer - - - - - - - - - PBSServer PBSServer.prog.com - - - - - - - - PBSServer - ux-gnn - - - - - - - - - - - - string STATIC resources_available.arch lin - - - - - - string STATIC resources_available.host ux- - - - - - - integer STATIC resources_available.mem 163 - - - - - - integer STATIC resources_available.ncpus 2 - - - - - - string STATIC resources_available.vnode ux- - - - - - - string STATIC resources_available.nodeclass 2CO

EVENT_TYPE ARG SERVER_NAME FQDN NODE_NAME VNODE VNODE_RANGE RESOURESOURCE_NAME RESOURCE_VALUE FROM_DATE TO_DATE COMMENT CUSTOM_OR_NSTALL_SNAPSHOT RANGE PBSServer - - - - - - - - - PBSServer PBSServer.prog.com - - - - - - - - PBSServer - ux-gnn - - - - - - - - - - - - string STATIC resources_available.arch lin - - - - - - string STATIC resources_available.host ux- - - - - - - integer STATIC resources_available.mem 163 - - - - - - integer STATIC resources_available.ncpus 2 - - - - - - string STATIC resources_available.vnode ux- - - - - - - string STATIC resources_available.nodeclass 2CO


18.5.4 Defining an execution host

The third record type that is necessary to describe a PBS Professional complex to PBSA is the NODE record. The NODE record describes the PBS Professional execution host. The fol-lowing information is necessary to define a NODE record and must be separated by a tab character:

• RANGE - Must be placed in the 2nd column


• NODE_NAME - Must be placed in the 5th column

• VNODE - Must be placed in the 6th column





PB 283


18

alid lues

Mandatory

R Ex. 1-3, 5-8 No

S Professional server. Yes

N Professional node for Yes

V No

F Date must be less than the TO_DATE

Yes

T Date must be greater than the FROM_DATE

Yes

Designates the current day

C SSUE No


.5.4.1 NODE record layout

Table 18-7: NODE record layout

Column Name

Column Number

Column Description

Data Type

VVa

ANGE 2 Defines a range of execution hosts (NODES) with like names and resources (homogenous).

String Integer range

ERVER_NAME 3 Server name as defined in PBS Professional

String Must be a valid PBS

ODE_NAME 5 Execution host name as defined in PBS Professional

String Must be a valid PBSSERVER_NAME

NODE 6 The natural vnode name. PBSA automatically creates a virtual node having the same name and resources as the execution host.

String Hyphen (-)

ROM_DATE 12 Date the event started Date YYYY-MM-DD

O_DATE 13 Date the event ended Date YYYY-MM-DD

String now

OMMENT 14 Descriptive reason for the event String Ex: HARDWARE_I



1

T APSHOT event is used to describe the c Professional server called PBSServer h s been up and running from January 1, 2

CE_DATATYPE RESOURCE_TYPE BUILT-IN FURNISHING_SERVERS - - - - - 2015-01-01 now - - - 2015-01-01 now - - -ux - - - custom -gnn - - - custom -84kb - - - custom - - - - custom -gnn - - - custom -RE1GB - - - custom -

84 PB

8.5.4.2 Sample NODE record

he highlighted record below is an example of a NODE record. In this example, the INSTALL_SNonfiguration of the current year’s PBS Professional complex. The PBS complex has a single PBSaving a single node called ux-gnn. The date range of the NODE record indicates that the node ha015 to “now” (currently is running).

#EVENT_TYPE ARG SERVER_NAME FQDN NODE_NAME VNODE VNODE_RANGE RESOURRESOURCE_NAME RESOURCE_VALUE FROM_DATE TO_DATE COMMENT CUSTOM_OR_INSTALL_SNAPSHOT RANGE PBSServer - - - - - - - -- - PBSServer PBSServer.prog.com - - - - - - -- - PBSServer - ux-gnn - - - - - -- - - - - - - string STATIC resources_available.arch lin- - - - - - - string STATIC resources_available.host ux-- - - - - - - integer STATIC resources_available.mem 163- - - - - - - integer STATIC resources_available.ncpus 2 - - - - - - - string STATIC resources_available.vnode ux-- - - - - - - string STATIC resources_available.nodeclass 2CO

PB 285


To E field to denote the node naming sch sk(*).

In three nodes having the same set of res

•

•

•

# CE_DATATYPE RESOURCE_TYPE R BUILT-IN FURNISHING_SERVERSI - - - - -- 2015-01-01 now - - -- 2015-01-01 now - - -- ux - - - custom -- gnn* - - - custom -- 84kb - - - custom -- - - - custom -- gnn* - - - custom -- RE1GB - - - custom -


describe a homogeneous range of nodes having similar names and like resources, use the RANGeme. When a range of nodes are being specified, the node name must be suffixed with an asteri

this example, RANGE is set to “1-3”. Using the pattern “ux-gnn*”, this describes the followingources:

node=ux-gnn1 virtual node=ux-gnn1



EVENT_TYPE ARG SERVER_NAME FQDN NODE_NAME VNODE VNODE_RANGE RESOURESOURCE_NAME RESOURCE_VALUE FROM_DATE TO_DATE COMMENT CUSTOM_OR_NSTALL_SNAPSHOT RANGE PBSServer - - - - - - - - - PBSServer PBSServer.prog.com - - - - - - - 1-3 PBSServer - ux-gnn* - - - - - - - - - - - - string STATIC resources_available.arch lin - - - - - - string STATIC resources_available.host ux- - - - - - - integer STATIC resources_available.mem 163 - - - - - - integer STATIC resources_available.ncpus 2 - - - - - - string STATIC resources_available.vnode ux- - - - - - - string STATIC resources_available.nodeclass 2CO


18.5.5 Defining a virtual node

The fourth record type that is necessary to describe a PBS Professional complex to PBSA is the VNODE record. The VNODE record describes the PBS Professional virtual node.

If virtual nodes are not explicitly defined for a node, PBS Professional creates a virtual node having the same name as the node (natural vnode). This is denoted on the server node record in the VNODE field.

Multiple virtual nodes may be defined for an execution host, therefore a VNODE record will be required for each virtual node. The following information is necessary to define a VNODE record and must be separated by a tab character.


• NODE_NAME - Must be placed in the 5th column

• VNODE - Must be placed in the 6th column

• VNODE_RANGE - Must be placed in the 7th column





PB 287


18

alid alues

Mandatory

S rofessional server. Yes

N rofessional execution host Yes

V rofessional virtual node for E_NAME

Yes

V x. 1-3, 5-8 No

F Date must be less than the TO_DATE

Yes

T Date must be greater than the FROM_DATE

Yes

Designates the current day

C

OWN

E

No


.5.5.1 VNODE record layout

Table 18-8: VNODE record layout

Column Name

ColumnNumber

Column Description

Data Type

VV

ERVER_NAME 3 Server name as defined in PBS Professional

String Must be a valid PBS P

ODE_NAME 5 Execution host name as defined in PBS Professional

String Must be a valid PBS Pfor SERVER_NAME

NODE 6 Virtual node name. String Must be a valid PBS PSERVER_NAME/NOD

NODE_RANGE 7 Defines a range of virtual nodes with like names and resources (homogeneous).

String Integer range E

ROM_DATE 12 Date the event started Date YYYY-MM-DD

O_DATE 13 Date the event ended Date YYYY-MM-DD

String now

OMMENT 14 Descriptive reason for the event String Ex:

SCHEDULE_SHUTD

HARDWARE_ISSUE

NETWORK_FAILUR



1

T NAPSHOT event is used to describe the c Professional server called PBSServer h tual node has been up and running from Ja

CE_DATATYPE RESOURCE_TYPE BUILT-IN FURNISHING_SERVERS - - - - - 2015-01-01 now - - - 2015-01-01 now - - -nux - - - custom --gnn - - - custom - - - - custom - - - - custom - - - - custom --gnn - - - custom -ORE0GB - - - custom - 2015-01-01 now - - -nux - - - custom - - - - custom --gnn - - - custom -384kb - - - custom - - - - custom -0mb - - - custom --gnn[0] - - - custom -ORE1GB - - - custom -

88 PB

8.5.5.2 Sample VNODE record

he highlighted record below is an example of a VNODE record. In this example, the INSTALL_Sonfiguration of the current year’s PBS Professional complex. The PBS complex has a single PBSaving a virtual node called ux-gnn[0]. The date range of the VNODE record indicates that the virnuary 1, 2015 to “now” (currently is running).

#EVENT_TYPE ARG SERVER_NAME FQDN NODE_NAME VNODE VNODE_RANGE RESOURRESOURCE_NAME RESOURCE_VALUE FROM_DATE TO_DATE COMMENT CUSTOM_OR_INSTALL_SNAPSHOT RANGE PBSServer - - - - - - - -- - PBSServer PBSServer.prog.com - - - - - - -- - PBSServer - ux-gnn - - - - - -- - - - - - - string STATIC resources_available.arch li- - - - - - - string STATIC resources_available.host ux- - - - - - - integer STATIC resources_available.mem 0b- - - - - - - integer STATIC resources_available.ncpus 0 - - - - - - - integer STATIC resources_available.vmem 0b- - - - - - - string STATIC resources_available.vnode ux- - - - - - - string STATIC resources_available.nodeclass 0C- - PBSServer - ux-gnn ux-gnn[0] - - - - -- - - - - - - string STATIC resources_available.arch li- - - - - - - integer STATIC resources_available.custom_res 3 - - - - - - - string STATIC resources_available.host ux- - - - - - - integer STATIC resources_available.mem 16- - - - - - - integer STATIC resources_available.ncpus 2 - - - - - - - integer STATIC resources_available.vmem 10- - - - - - - string STATIC resources_available.vnode ux- - - - - - - string STATIC resources_available.nodeclass 2C

PB 289


To VNODE_RANGE field to denote the vir

W sterisk enclosed in square brackets ([*]).

In following three virtual nodes having the sam

•

•

•

# CE_DATATYPE RESOURCE_TYPE R BUILT-IN FURNISHING_SERVERSI - - - - -- 2015-01-01 now - - -- 2015-01-01 now - - -- nux - - - custom -- -gnn - - - custom -- - - - custom -- - - - custom -- - - - custom -- -gnn - - - custom -- ORE0GB - - - custom -- 2015-01-01 now - - -- nux - - - custom -- - - - custom -- -gnn - - - custom -- 384kb - - - custom -- - - - custom -- 0mb - - - custom -- -gnn[*] - - - custom -- ORE1GB - - - custom -


describe a homogeneous range of virtual nodes having similar names and like resources, use thetual node naming scheme.

hen a range of virtual nodes are being specified, the virtual node name must be suffixed with an a

this example, VNODE_RANGE is set to “0-2”. Using the pattern “ux-gnn[*]”, this describes thee set of resources:

ux-gnn[0]

ux-gnn[1]

ux-gnn[2]

EVENT_TYPE ARG SERVER_NAME FQDN NODE_NAME VNODE VNODE_RANGE RESOURESOURCE_NAME RESOURCE_VALUE FROM_DATE TO_DATE COMMENT CUSTOM_OR_NSTALL_SNAPSHOT RANGE PBSServer - - - - - - - - - PBSServer PBSServer.prog.com - - - - - - - - PBSServer - ux-gnn - - - - - - - - - - - - string STATIC resources_available.arch li - - - - - - string STATIC resources_available.host ux - - - - - - integer STATIC resources_available.mem 0b - - - - - - integer STATIC resources_available.ncpus 0 - - - - - - integer STATIC resources_available.vmem 0b - - - - - - string STATIC resources_available.vnode ux - - - - - - string STATIC resources_available.nodeclass 0C - PBSServer - ux-gnn ux-gnn[*] 0-2 - - - - - - - - - - string STATIC resources_available.arch li - - - - - - integer STATIC resources_available.custom_res 3 - - - - - - string STATIC resources_available.host ux - - - - - - integer STATIC resources_available.mem 16 - - - - - - integer STATIC resources_available.ncpus 2 - - - - - - integer STATIC resources_available.vmem 10 - - - - - - string STATIC resources_available.vnode ux - - - - - - string STATIC resources_available.nodeclass 2C


18.5.6 Defining resources

The fifth record type that is necessary to describe a PBS Professional complex to PBSA is the RESOURCE record. The RESOURCE record describes the resources available on a PBS Professional execution host or virtual node. The following information is necessary to define a RESOURCE record and must be separated by a tab character:

• RESOURECE_DATATYPE - Must be placed in the 8th column

• RESOURCE_TYPE - Must be placed in the 9th column

• RESOURCE_NAME - Must be placed in the 10th column

• RESOURCE_VALUE - Must be placed in the 11th column

• CUSTOM_OR_BUILT_IN - Must be placed in the 14th column


PB 291


18

Valid Values

Mandatory

R Yes

R Yes

R id PBS Professional Yes

R ma separated list of es.

rce vailable.host” the value e the node name.

rce vailable.vnode” the value e the virtual node name.

Yes

C y the value “custom” is Yes


.5.6.1 RESOURCE record layout

Table 18-9: RESOURCE record layout

Column Name

Column Number

Column Description

Data Type

ESOURECE_DATATYPE 8 Datatype of resource String integer

string

ESOURCE_TYPE 9 Describes the type of resourcel String DYNAMIC

10 STATIC

ESOURCE_NAME 11 Name of resource as defined in PBS Professional

String Must be a valresource.

ESOURCE_VALUE 12 Value of resource as defined in PBS Professional

String Can be a comresource valu

For the resou“resources_awill always b

For the resou“resources_awill always b

USTOM_OR_BUILT_IN 14 Defines whether this resource is a custom resource or a built-in resource. Currently, only custom resources are supported.

String Currently onlsupported.



1

T LL_SNAPSHOT event is used to d a single PBS Professional server called P available on the node and virtual node.

M

CE_DATATYPE RESOURCE_TYPE BUILT-IN FURNISHING_SERVERS - - - - - 2015-01-01 now - - - 2015-01-01 now - - -nux - - - custom --gnn - - - custom - - - - custom - - - - custom - - - - custom --gnn - - - custom -ORE0GB - - - custom - 2015-01-01 now - - -nux - - - custom - - - - custom --gnn - - - custom -384kb - - - custom - - - - custom -0mb - - - custom --gnn[0] - - - custom -ORE1GB - - - custom -

led Abaqus,Radioss - - - custom -

92 PB

8.5.6.2 Sample RESOURCE record

he highlighted records below are examples of a RESOURCE record. In this example, the INSTAescribe the configuration of the current year’s PBS Professional complex. The PBS complex has BSServer having a virtual node called ux-gnn[0]. The RESOURCE records describes the resources

ultiple resource values can be specified as comma separated list:

#EVENT_TYPE ARG SERVER_NAME FQDN NODE_NAME VNODE VNODE_RANGE RESOURRESOURCE_NAME RESOURCE_VALUE FROM_DATE TO_DATE COMMENT CUSTOM_OR_INSTALL_SNAPSHOT RANGE PBSServer - - - - - - - -- - PBSServer PBSServer.prog.com - - - - - - -- - PBSServer - ux-gnn - - - - - -- - - - - - - string STATIC resources_available.arch li- - - - - - - string STATIC resources_available.host ux- - - - - - - integer STATIC resources_available.mem 0b- - - - - - - integer STATIC resources_available.ncpus 0 - - - - - - - integer STATIC resources_available.vmem 0b- - - - - - - string STATIC resources_available.vnode ux- - - - - - - string STATIC resources_available.nodeclass 0C- - PBSServer - ux-gnn ux-gnn[0] - - - - -- - - - - - - string STATIC resources_available.arch li- - - - - - - integer STATIC resources_available.custom_res 3 - - - - - - - string STATIC resources_available.host ux- - - - - - - integer STATIC resources_available.mem 16- - - - - - - integer STATIC resources_available.ncpus 2 - - - - - - - integer STATIC resources_available.vmem 10- - - - - - - string STATIC resources_available.vnode ux- - - - - - - string STATIC resources_available.nodeclass 2C

- - - - - - - integer STATIC resources_available.pas_applications_enab

PB 293


18


.5.7 Install snapshot examples



T another node coming online April 1.

- - - 5-01-01 2015-12-31 - - -5-01-01 2015-12-31 - - - - - - custom - - - - custom - - - - custom - - - - custom - - - - custom - GB - - - custom - - - - 5-01-01 2015-12-31 - - -5-01-01 2015-12-31 - - - - - - custom - - - - custom - - - - custom - - - - custom - - - - custom - GB - - - custom - 5-04-01 2015-12-31 - - - - - custom - - - - custom - - - - custom - - - - custom - - - - custom - GB - - - custom -

94 PB

his example shows a complex with a server having 8 nodes that came online January 1, 2015 with

INSTALL_SNAPSHOT RANGE blrhpc3 - - - - - - - - - -- - blrhpc3 hp.prog.com - - - - - - - 201- 1-8 blrhpc3 - blrhpc3n* - - - - - - 201- - - - - - - string STATIC resources_available.arch linux - - - - - - - string STATIC resources_available.host blrhpc3n* - - - - - - - string STATIC resources_available.mem 32900260 - - - - - - - string STATIC resources_available.npus 12 - - - - - - - string STATIC resources_available.vnode blrhpc3n* - - - - - - - string STATIC resources_available.nodeclass 1CORE4INSTALL_SNAPSHOT RANGE blrhpc3 - - - - - - - - - -- - blrhpc3 hp.prog.com - - - - - - - 201- 1-8 blrhpc3 - blrhpc3n* - - - - - - 201- - - - - - - string STATIC resources_available.arch linux - - - - - - - string STATIC resources_available.host blrhpc3n* - - - - - - - string STATIC resources_available.mem 32900260 - - - - - - - string STATIC resources_available.npus 12 - - - - - - - string STATIC resources_available.vnode blrhpc3n* - - - - - - - string STATIC resources_available.nodeclass 1CORE4- - blrhpc3 - blrhpc3n30 - - - - - - 201- - - - - - - string STATIC resources_available.arch linux - - - - - - - string STATIC resources_available.host blrhpc3n30 - - - - - - - string STATIC resources_available.mem 32900260 - - - - - - - string STATIC resources_available.npus 12 - - - - - - - string STATIC resources_available.vnode blrhpc3n30- - - - - - - string STATIC resources_available.nodeclass 1CORE4

PB 295


Th rtual nodes that came online January 1st, 20

IN - - - - 5-01-01 2015-12-31 - - -- 5-01-01 2015-12-31 - - -- - - - custom -- - - - custom -- - - - custom -- - - - custom -- ] - - - custom - - GB - - - custom - - -01-01 2015-12-31 - - -- - - - custom -- - - - custom -- - - - custom -- - - - custom -- ] - - - custom - - GB - - - custom -


is example shows that a complex has a single server with a single node that was divided into 4 vi15. The virtual nodes are called blrhpc3n1v0, blrhpc3n1v1, blrhpc3n1v2, and blrhpc3n1v9.

STALL_SNAPSHOT RANGE blrhpc3 - - - - - - - - - - - blrhpc3 hp.prog.com - - - - - - - 201 - blrhpc3 - blrhpc3n1 blrhpc3n1v[*] 0-2 - - - - 201 - - - - - - string STATIC resources_available.arch linux - - - - - - string STATIC resources_available.host blrhpc3n1 - - - - - - string STATIC resources_available.mem 32900260 - - - - - - string STATIC resources_available.npus 12 - - - - - - string STATIC resources_available.vnode blrhpc3nv[* - - - - - - string STATIC resources_available.nodeclass 1CORE4 - blrhpc3 - blrhpc3n1 blrhpc3n1v[9] - - - - - 2015 - - - - - - string STATIC resources_available.arch linux - - - - - - string STATIC resources_available.host blrhpc3n1 - - - - - - string STATIC resources_available.mem 32900260 - - - - - - string STATIC resources_available.npus 12 - - - - - - string STATIC resources_available.vnode blrhpc3nv[9 - - - - - - string STATIC resources_available.nodeclass 1CORE4



T

- - - -01-01 2015-12-31 - - -

5-01-01 2015-12-31 - - - - - - custom - - - - custom - - - - custom - GB - - - custom - 2015-01-01 2015-12-31 - - -

- - - custom - - - - custom - - - - custom -GB - - - custom - 2015-01-01 2015-12-31 - - -

- - - custom - - - - custom - - - - custom -GB - - - custom -

96 PB

his example shows a range of virtual nodes within a range of nodes:

INSTALL_SNAPSHOT RANGE blrhpc3 - - - - - - - - - -- - blrhpc3 hp.prog.com - - - - - - - 2015- 1-3 blrhpc3 - blrhpc3n* - - - - - - 201- - - - - - - string STATIC resources_available.arch linux - - - - - - - string STATIC resources_available.mem 32900260 - - - - - - - string STATIC resources_available.npus 12 - - - - - - - string STATIC resources_available.nodeclass 1CORE4- - blrhpc3 - blrhpc3n1 abc[*] 0-10 - - - -

- - - - - - - string STATIC resources_available.arch linux - - - - - - - string STATIC resources_available.mem 32900260 - - - - - - - string STATIC resources_available.npus 12 - - - - - - - string STATIC resources_available.nodeclass 1CORE4- - blrhpc3 - blrhpc3n2 abc[*] 0-10 - - - -

- - - - - - - string STATIC resources_available.arch linux - - - - - - - string STATIC resources_available.mem 32900260 - - - - - - - string STATIC resources_available.npus 12 - - - - - - - string STATIC resources_available.nodeclass 1CORE4

PB 297


Th called “blrhpcbkup”. .

18

Th ber 1st and will continue to remain on-lin

IN - - - -

- 1-01 2015-12-31 - - -

- 1-01 2015-12-31 - - -

- - - - custom -- - - - custom -- - - - custom -- - - - custom -- - - - custom - - GB - - - custom -

A - - -

- 2015-11-01 now - - -

- - - - custom -- - - - custom -- - - - custom -- GB - - - custom -


is example shows that this complex has a single server with a secondary server (failover server)

.5.8 Adding a node example

is example shows how to add new nodes to the complex. The nodes will come on-line on Noveme, unless a DOWN or DELETE record is added to the node details file.

STALL_SNAPSHOT RANGE blrhpc3-alias-blrhpcbkup - - - - - - - - -

- blrhpc3-alias-blrhpcbkup hp.prog.com - - - - - - - 2015-0

- blrhpc3 - blrhpc3n30 - - - - - - 2015-0

- - - - - - string STATIC resources_available.arch linux - - - - - - string STATIC resources_available.host blrhpc3n30 - - - - - - string STATIC resources_available.mem 32900260 - - - - - - string STATIC resources_available.npus 12 - - - - - - string STATIC resources_available.vnode blrhpc3n30 - - - - - - string STATIC resources_available.nodeclass 1CORE4

DD RANGE - - - - - - - - - - -

1-10 blrhpc3 - blrhpc3n* - - - - - -

- - - - - - string STATIC resources_available.arch linux - - - - - - string STATIC resources_available.mem 32900260 - - - - - - string STATIC resources_available.npus 12 - - - - - - string STATIC resources_available.nodeclass 1CORE4



1

T

A

A

U

- - -

2015-01-01 now - - -


- - -

2015-11-01 2015-11-15 - - -


98 PB

8.5.9 Bring a node down example

his example shows how to bring a node down.

node was added on January 1st, 2015.

t a later date it is brought down for maintenance from November 1st through November 15th.

se the ADD event to make a node active after a DOWN event.

ADD RANGE - - - - - - - - - - -

- - blrhpc3 - blrhpc3n30 - - - - - -


DOWN RANGE - - - - - - - - - - -

- - blrhpc3 - blrhpc3n30 - - - - - -


PB 299


18

Th

Th

It i

A - - -

- 015-01-01 now - - -


D - - -

- 015-11-01 now - - -



.5.10 Deleting a node example

is example shows how to delete a node.

e node is added on January 1st, 2015.

s then deleted on November 1st, 2015.

DD RANGE - - - - - - - - - - -

- blrhpc3 - blrhpc3n30 - - - - - - 2


ELETE RANGE - - - - - - - - -

- blrhpc3 - blrhpc3n30 - - - - - - 2


Chapter 19

Advanced Configuration

19.1 Disabling PBSA service ports

19.1.1 Disabling the HTTP port

Disabling the PBSA HTTP service port requires root privileges.

To disable the PBSA HTTP service port so that access to PBSA is done exclusively over https, follow these steps:

1. Stop the PBSA service - see section 14.1.2, “Stopping the PBSA service”, on page 142.

2. Edit the file server.xml. The location of the file is::

INSTALL_DIR_13.0/portal/thirdparty/apache/tomcat/conf


Chapter 19 Advanced Configuration

3. Replace the following XML:

with this XML:

Please note that the ports “9000” and “9143” are the default http and https ports used by PBSA, and may be different than the ports that are being used at your site. Please see section 3.3.5, “Ports used by PBSA”, on page 18 for more information about the selection of communication ports by PBSA during installation.

4. Start the PBSA service -see section 14.1.1, “Starting the PBSA service”, on page 141.

<Connector connectionTimeout="20000" port="9000" protocol="HTTP/1.1" redirectPort="9143"/>

<Connector address="localhost" connectionTimeout="20000" port="9000" protocol="HTTP/1.1" redirectPort="9143"/>


Advanced Configuration Chapter 19

19.1.2 Disabling the HTTPS port

Disabling the PBSA HTTPS service port requires root privileges.

To disable the PBSA HTTPS service port so that access to PBSA is done exclusively over http, follow these steps:

1. Stop the PBSA service - see section 14.1.2, “Stopping the PBSA service”, on page 142.



3. Comment out the https connector by adding '' around this section of XML:

The XML section should look like this :

Please note that the ports “9000” and “9143” are the default http and https ports used by PBSA, and may be different than the ports that are being used at your site. Please see section 3.3.5, “Ports used by PBSA”, on page 18 for more information about the selection of communication ports by PBSA during installation.

4. Start the PBSA service -see section 14.1.1, “Starting the PBSA service”, on page 141.

<Connector port="9143" maxThreads="200" scheme="https" secure="true" SSLEnabled="true" clientAuth="false" sslProtocols="TLSv1, TLSv1.1, TLSv1.2" keystoreFile="${HWE_INSTALLATION_DIRECTORY}/config/ams.keystore" keystorePass="changeit"/>


19.2 Configuring logging

Logging behavior can be defined in the file log4j.properties. The location of the file is::

INSTALL_DIR_13.0/portal/config

By default, PBSA logs at an "INFO" level, generates a new log file once the size of the cur-rent log file exceeds 100MB, and keeps a maximum of 1000 files as backup.

19.2.1 log4j.properties file

19.2.2 Configuring logging behavior

To configure the level of logging:

Edit the log4j.properties file and set the parameter log4j.rootLogger to one of the fol-lowing values:

• OFF - designated the highest possible ranking and is intended to turn off logging

• FATAL - designates very severe error events that will presumably lead the application to abort

log4j.appender.frameworkRollingFile=org.apache.log4j.RollingFileAppenderlog4j.appender.frameworkRollingFile.File=${HWE_INSTALLATION_DIRECTORY}/logs/pbsworks.loglog4j.appender.frameworkRollingFile.MaxFileSize=100MBlog4j.appender.frameworkRollingFile.MaxBackupIndex=1000log4j.appender.frameworkRollingFile.layout=org.apache.log4j.PatternLayoutlog4j.appender.frameworkRollingFile.layout.ConversionPattern= %n%p, %d{dd MMM yyy HH:mm:ss,SSS} -{%F: %L} - %m %nlog4j.rootLogger=INFO, frameworkRollingFile

log4j.appender.KeyPerformanceAppender=org.apache.log4j.RollingFileAppenderlog4j.appender.KeyPerformanceAppender.File=keyPerFormanceLog.csvlog4j.appender.KeyPerformanceAppender.MaxFileSize=100MBlog4j.appender.KeyPerformanceAppender.MaxBackupIndex=1000log4j.appender.KeyPerformanceAppender.layout=org.apache.log4j.PatternLayoutlog4j.appender.KeyPerformanceAppender.layout.ConversionPattern=%m%n

log4j.logger.KeyPerformanceLogger=INFO, KeyPerformanceAppenderlog4j.additivity.KeyPerformanceLogger=false



• ERROR - designates error events that might still allow the application to continue run-ning

• WARN - designates potentially harmful situations

• INFO - designates informational messages that highlight the progress of the application at coarse-grained level

• DEBUG - designates fine-grained informational events that are most useful to debug an application

• TRACE - designates finer-grained informational events than the DEBUG

• ALL - designates the lowest possible rank and is intended to turn on all logging

The PBSA service will need to be restarted (see section 14.1.3, “Restarting the PBSA ser-vice”, on page 143) for the changes to be reflected.

19.2.3 Configuring log file size and number of backups

To define the maximum log file size and the number of backup files:

Edit the log4j.properties file and set the MaxFileSize and MaxBackupIndex parame-ters:


log4j.logger.KeyPerformanceLogger=INFO, KeyPerformanceAppender

log4j.appender.frameworkRollingFile.MaxFileSize=100MBlog4j.appender.frameworkRollingFile.MaxBackupIndex=1000



19.3 Configuring the license server

19.3.1 Initial license server configuration

If you installed PBSA without providing a license server, then prior to starting the PBSA ser-vice for the first time, you must configure PBSA with a valid license server. To configure the license server, modify the factorySettings.xml file. The location of this file is:

INSTALL_DIR_13.0/portal/binaries/framework/data:

<?xml version="1.0" encoding="UTF-8"?><Settings> <Category name="Root"> <Category name="General"> <Setting name="savesession" preference="true"> <value>true</value> <class>com.altair.awpf.services.preferences.model.BooleanPreference</class> <displayName>Save Session</displayName> <isEnforced>false</isEnforced> <private>false</private> </Setting> <Setting name="notificationpanelstate" preference="false"> <value>hide</value> <class>com.altair.awpf.services.preferences.model.Setting</class> <private>true</private> </Setting> </Category> <Category name="License Server"> <Setting name="Location" preference="true"> <value></value> <class>com.altair.awpf.services.preferences.model.StringPreference</class> <displayName>Server Location</displayName> <isEnforced>true</isEnforced> <private>false</private> </Setting> </Category> </Category></Settings>



19.3.1.1 Configuring the license server

Edit the factorySettings.xml file and add the <port@hostname> of the license server.

Now you may start the PBSA service (see section 14.1.1, “Starting the PBSA service”, on page 141).

<Category name="License Server"> <Setting name="Location" preference="true"> <value>port@hostname</value> <class>com.altair.awpf.services.preferences.model.StringPreference</class> <displayName>Server Location</displayName> <isEnforced>true</isEnforced> <private>false</private> </Setting></Category>



19.3.2 Configuring the license server after starting PBSA

If the PBSA service has been started, and either a valid license server has not been configured for PBSA, or if the license server just needs to be changed, then modify the adminSet-tings.xml file. The location of this file is:

INSTALL_DIR_13.0/portal/binaries/framework/data

19.3.2.1 adminSettings.xml file:

<?xml version="1.0" encoding="UTF-8"?><Settings> <Category name="Root"> <Category name="General"> <Setting name="savesession" preference="true"> <value>true</value> <class>com.altair.awpf.services.preferences.model.BooleanPreference</class> <displayName>Save Session</displayName> <isEnforced>false</isEnforced> <private>false</private> </Setting> <Setting name="notificationpanelstate"> <value>hide</value> <class>com.altair.awpf.services.preferences.model.Setting</class> <isEnforced>false</isEnforced> <private>true</private> </Setting> </Category> <Category name="License Server"> <Setting name="Location" preference="true"> <value>6200@trlicsrv03</value> <class>com.altair.awpf.services.preferences.model.StringPreference</class> <displayName>Server Location</displayName> <isEnforced>true</isEnforced> <private>false</private> </Setting> </Category> </Category></Settings>



19.3.2.2 Configuring the license server

Edit the adminSettings.xml file and change the <port@hostname> of the license server.


<Category name="License Server"> <Setting name="Location" preference="true"> <value>port@hostname</value> <class>com.altair.awpf.services.preferences.model.StringPreference</class> <displayName>Server Location</displayName> <isEnforced>true</isEnforced> <private>false</private> </Setting></Category>



19.4 Configuring JVM Performance

19.4.1 Configuring JVM of PBSA service

The Java Virtual Machine (JVM) heap size of the PBSA service may need to be adjusted.

The default JVM heap size is based on the licensing option:

• Base - 4 GB

• Standard - 8 GB

• Advanced - 16 GB

If OutOfMemory errors or warnings are reported in the log files, this value should be increased. It can be increased in multiples of 256 MB. The Java heap size default is stored in a script file called memoryRecommendation.sh. The location of this is:

INSTALL_DIR_13.0/portal/scripts

This is the area of the file that will need to be modified:

For example, to change the Java heap size for a Base licensing option from “4096” to “8192”, change the following line:

to:

The PBSA service will need to be restarted (see section 14.1.3, “Restarting the PBSA ser-vice”, on page 143) for the changes to be reflected after configuring the JVM.

if [ "$numberOfUnits" -eq "10000" ]; then echo "4086 Base"elif [ "$numberOfUnits" -eq "30000" ];then echo "8192 Standard"elif [ "$numberOfUnits" -eq "60000" ]then echo "16384 Advance"fi

if [ "$numberOfUnits" -eq "10000" ]; then echo "4086 Base"

if [ "$numberOfUnits" -eq "10000" ]; then echo "8192 Base"



19.4.2 Configuring JVM of PBSA Data Collector

The Java Virtual Machine (JVM) heap size of the PBSA Data Collector may need to be adjusted. Currently, the default JVM heap size is 256 MB. If OutOfMemory errors or warn-ings are reported in the log files, this value should be increased. It can be increased in multi-ples of 256 MB.

The Java heap size default is stored in a script file called pbsworks-dc-start.sh. The location of this file is:

INSTALL_DIR_13.0/dc/scripts

To increase the JVM maximum memory limit, modify the -Xmx256m argument by increasing the value "256”.

To change the JVM minimum memory limit, modify the -Xms64m argument by increasing or decreasing the value “64”.

This is the area of the file that will need to be modified (this file is too large to show in its entirety):

For instance, JAVA_OPTS="-Xms64m -Xmx256m", allocates a minimum of 64 MB and a maximum of 256 MB of JVM.

The PBSA Data Collector service will need to be restarted (see section 14.2.3, “Restarting the PBSA Data Collector service”, on page 145) for the changes to be reflected after configuring the JVM.

JAVA_OPTS="-Xms64m -Xmx256m"



19.5 Configuring SSH Authentication

During installation, PBSA can be configured to use SSH based authentication. This authenti-cation information is stored in the file AA_jaas.config. The location of the file is:

INSTALL_DIR_13.0/portal/config/AA

19.5.1 AA_jaas.config

19.5.2 Changing the SSH server

Edit the AA_jaas.config file and change host=<NEW_SSH_SERVER> to the new SSH server IP address or hostname.


LoginModule { com.altair.jaas.module.SSHUnixLoginModule required host="localhost" port="22" cache_ttl="60" debug=false;};

host="<NEW_SSH_SERVER>"



19.6 Changing the accounting log location

These instructions pertain to installations of PBSA where the PBSA Data Collector has been installed on a separate server than the Web Application and Parser. During installation of the PBSA Data Collector, the location of the PBS Professional accounting logs is provided. This information can be changed without reinstalling the Data Collector by editing the file con-fig.xml. The location of the file is:

INSTALL_DIR_13.0/dc/modules/pbsa/config/

19.6.1 config.xml

Edit the config.xml file and change the value of the accounting-log-dir to the correct PBS Professional accounting log directory.

The PBSA Data Collector will need to be restarted (see section 14.2.3, “Restarting the PBSA Data Collector service”, on page 145) for the changes to be reflected.

<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">

<properties> <comment>PBSA Configuration File</comment> <entry key="accounting-log-dir">pathToAcctLogs</entry></properties>

<entry key="accounting-log-dir">/var/spool/PBS/server_priv/accounting</entry>


19.7 Configuring communication between the DC and the PBSA service

These instructions pertain to installations of PBSA where the PBSA Data Collector has been installed on a separate server than the Web Application.

The server URL over which the Data Collector communicates to the PBSA service can be configured or changed without reinstalling the Data Collector. To configure or change the server URL, edit the file config.xml. The location of the file is:

INSTALL_DIR_13.0/dc/config/

19.7.1 config.xml

19.7.2 Changing the server URL

Edit the config.xml file and change the value of the server-url to the correct host-name:port of the machine where the PBSA Web Application and Parser were installed.

The PBSA Data Collector will need to be restarted (see section 14.2.3, “Restarting the PBSA Data Collector service”, on page 145) for the changes to be reflected.

<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">

<properties> <comment>DC Configuration file</comment>

<entry key="server-url">https://hwesuse113-01:9343</entry> <entry key="service-context">/parser/dc</entry> <entry key="upload-context">/parser/upload</entry> <entry key="server-name">DEFAULT</entry>

 <entry key="transport-interval">10</entry> <entry key="data-transport">com.altair.pbsworks.dc.transport.http.HTTPFileTransport</entry></properties>

<entry key="server-url">https://newhost:newport</entry>


19.8 Configuring plot point decimation

Charts may have an unnecessary number of data points relative to the number of pixels that can be displayed by a browser or a desktop application. To technically improve the rendering of charts within the web browser, and to provide usable charts that show trends and patterns, PBSA uses a decimation or downsampling strategy to reduce the number of plot points dis-played in a chart.

Note 1: When a chart is plotted using the decimated data subset, it will differ from the exported dataset which will contain the complete data.

Note 2: All statistical calculations performed on the plotted chart is based on the decimated data and not on the full available data.

Configurable limitations are set for both the x and y coordinates that essentially filter the data that is plotted on the graph. This information is stored in a file called pbsaconfig.prop-erties.

The location of this file is:

INSTALL_DIR_13.0/portal/services/pbsa/config

PBSA is configured by default to display at most 150 points on the x-axis, and 50 points on the y-axis in the case of stacked and xyz charts.

19.8.1 Changing the decimation defaults

To change the default decimation values, edit the pbsaconfig.properties file and change the value of the decimation defaults for the x and y coordinates.

If the PBSA service is currently running, then it will need to be restarted (see section 14.1.3, “Restarting the PBSA service”, on page 143) for the changes to be reflected.

Changing these defaults can lead to a slowdown in chart rendering, however if you have a need to analyze the chart data, you may use the PBSA Web Application to export the data to an excel spreadsheet for further inspection and analysis of the data.

#Decimation Propertiesdecimation=truedecimate-x=150decimate-y=50



19.9 Changing the license option

During installation of PBSA, there are three licensing options available based on the number of PBS Professional accounting log job records that will be parsed and analyzed (HWU is Hyper Works Units):




For licensing, a job record is not equivalent to a PBS Professional job accounting log "E" record. There is one “licensing record” for every “job node”. For example, a single job that runs on 100 nodes equates to 100 records for licensing. Ten jobs that each use four (4) nodes equates to 40 licensing records. In addition, if a site adds additional data to the PBS Analytics database (not related to jobs), that data is also counted as far as licensing is concerned. For example, if a site adds "node temperature" once per day on 256 nodes, that would add 256 * 365 = 93,440 license records per year.

Over time the licensing option may need to be changed to accommodate an increase in the number of accounting log job records. If the licensing option is not upgraded to accommo-date the increased accounting log activity, then the PBSA Web Application will display the most recent accounting log data up to the amount of data that it is licensed for, effectively fil-tering out any data that exceeds the licensing limit (older data).

Upon upgrading the license option, all data will become visible, unless the accounting log activity exceeds the newest licensing option.

Upgrading the license option can be done by modifying the license.xml file. The loca-tion of this file is:

INSTALL_DIR_13.0/portal/binaries/framework/services/config

In addition to the license.xml file, there are three other licensing files within this direc-tory. A file for each type of licensing option:

• license_advanced.xml

• license_base.xml

• license_standard.xml

To change the licensing option, simply copy the appropriate licensing file to the license.xml file.

If the PBSA service is currently running, then it will need to be restarted (see section 14.1.3, “Restarting the PBSA service”, on page 143) for the changes to be reflected.



19.10 Configuring a site specific SSL certificate

Configuring a site specific SSL certificate requires root privileges.

If your site would like to generate a keystore file specifically for PBSA, see section 19.11, “Creating a site specific SSL keystore file”, on page 318 for more information.

To configure PBSA to use a site specific SSL certificate, rather than the default SSL certifi-cate installed with PBSA, follow these steps:

1. Stop the PBSA service -see section 14.1.2, “Stopping the PBSA service”, on page 142.

2. Copy your site’s keystore file to:

INSTALL_DIR_13.0/portal/config



4. Replace the default keystore filename and password with the filename and password of your site’s keystore file. Replace this XML:

with this XML:

Please note that the port “9143” is the default https port used by PBSA, and may be dif-ferent than the port that are being used at your site. Please see section 3.3.5, “Ports used by PBSA”, on page 18 for more information about the selection of communication ports by PBSA during installation.

5. Start the PBSA service -see section 14.1.1, “Starting the PBSA service”, on page 141

<Connector port="9143" maxThreads="200" scheme="https" secure="true" SSLEnabled="true" clientAuth="false" sslProtocol="TLS" keystoreFile="${HWE_INSTALLATION_DIRECTORY}/config/ams.keystore" keystorePass="changeit"/>

<Connector port="9143" maxThreads="200" scheme="https" secure="true" SSLEnabled="true" clientAuth="false" sslProtocol="TLS" keystoreFile="${HWE_INSTALLATION_DIRECTORY}/config/your_keystore_filename" keystorePass="your_keystore_password"/>



19.11 Creating a site specific SSL keystore file

Follow the below steps to create a site specific SSL keystore file:

1. Determine if the environment variable JAVA_HOME is set properly by entering the following at the Linux command line prompt:

echo $JAVA_HOME

2. JAVA_HOME should be set to INSTALL_DIR_13.0/portal/thirdparty/java. If it is not then enter the following command:

export JAVA_HOME=INSTALL_DIR_13.0/portal/thirdparty/java

3. Create a Tomcat certificate keystore file by running the following command:

$JAVA_HOME/bin/keytool -keystore INSTALL_DIR_13.0/portal/config/key-

store -genkey -alias pbsworks -keyalg RSA

4. You will be prompted for the keystore password. The default password used by Tom-cat is "changeit" (all lower case), although you can specify a custom password if you like. Make a note of this password since it will be required in later steps of the SSL configuration.

5. Next, you will be prompted for general information about this Certificate, such as company, contact name, and so on. This information will be displayed to users who attempt to access a secure page in your application, so make sure that the information provided here matches what they will expect.

6. Finally, you will be prompted for the key password, which is the password specifi-cally for this certificate (as opposed to any other certificates stored in the same key-store file). You MUST use the same password here as was used for the keystore password itself. This is a restriction of the Tomcat implementation. (Currently, the keytool prompt will tell you that pressing the ENTER key does this for you automat-ically.)

If everything was successful, you now have a keystore file in the INSTALL_DIR_13.0/portal/config directory with a certificate that can be used by your server.



19.12 Changing the ports used by PBSA

19.12.1 Changing the HTTP port of the PBSA service

Changing the HTTP port of the PBSA service requires root privileges.

Stop the PBSA and all Data Collector services - see Chapter 14, "PBSA service commands", on page 141.

Two files require modification. The first file is:

INSTALL_DIR_13.0/portal/services/pbsa/config/pbsaconfig.properties

Change the port number for the tags:

• connectionCache

• Server

The second file that requires modification is:

INSTALL_DIR_13.0/portal/thirdparty/apache/tomcat/conf/server.xml

Change the port number of the HTTP connector for the PBSA service:

Start the PBSA and all Data Collector services - see Chapter 14, "PBSA service commands", on page 141.

Please note that the port “9000” is the default HTTP port used by PBSA, and may be different than the port that is being used at your site. Please see section 3.3.5, “Ports used by PBSA”, on page 18 for more information about the selection of communication ports by PBSA during installation.

#Cache PropertiesconnectionCache=http://localhost:9000/altairanalytics/servlet/FlushCache

#OLAP4j Connection PropertiesServer=http://localhost:9000/altairanalytics/xmla

<Connector connectionTimeout="20000" port="9000" protocol="HTTP/1.1" redirectPort="9143"/>



19.12.2 Changing the HTTPS port of the PBSA service

Changing the HTTPS port of the PBSA service requires root privileges.


The following file requires modification:


Change the port number of the HTTPS connector for the PBSA service:


Please note that the port “9143” is the default HTTPS port used by PBSA, and may be differ-ent than the port that is being used at your site. Please see section 3.3.5, “Ports used by PBSA”, on page 18 for more information about the selection of communication ports by PBSA during installation.

<Connector port="9143" maxThreads="200" scheme="https" secure="true" SSLEnabled="true" clientAuth="false" sslProtocol="TLS" keystoreFile="${HWE_INSTALLATION_DIRECTORY}/config/ams.keystore" keystorePass="changeit"/>



19.12.3 Changing the port of the PBSA Data Collector service

Changing the HTTP port of the PBSA Data Collector service requires root privileges.


Two files require modification. The first file is:

INSTALL_DIR_13.0/dc/config/config.xml

Change the port number for the tag:

• entry key=”server-url”

The second file that requires modification is:


Change the port number of the HTTP connector for the Data Collector service:


Please note that the port “9343” is the default HTTP port used by the PBSA Data Collectors, and may be different than the port that is being used at your site. Please see section 3.3.5, “Ports used by PBSA”, on page 18 for more information about the selection of communica-tion ports by PBSA during installation.

<entry key="server-url">https://localhost:9343</entry>

<Connector port="9343" maxThreads="200" scheme="https" secure="true" SSLEnabled="true" clientAuth="false" sslProtocol="TLS" keystoreFile="${HWE_INSTALLATION_DIRECTORY}/services/pbsa/config/pbsa.keystore" keystorePass="changeit"/>


Chapter 20

Reparsing the accounting logsUse the PBSA dataset reset command to discard all parsed accounting log data including the history files indicating which PBS Professional accounting log files have already been trans-ported to the machine where the Parser is installed. As of PBSA 13.0, the pbsa-data-reset will also automatically reset all Data Collectors.

1. Stop the PBSA service by issuing the command:

/etc/init.d/pbsworks-pbsa stop

2. Execute the following command as the root user on the machine where the Web Appli-cation and Parser are installed.

pbsa-data-reset

3. The data reset command resets the PBSA database password to “pbsworks@123”. Use the pbsa-data-password command to change it to a site specific password:

pbsa-data-password OLD_PASSWORD NEW_PASSWORD

4. Modify the PBSA configuration if necessary. For more information see section 8.5, “Configuring PBSA”, on page 108.

5. Initiate the parsing cycle by starting the PBSA service by issuing the command:



Chapter 20 Reparsing the accounting logs


Chapter 21

Disaster RecoveryThis chapter provides guidance on how to backup and restore PBS Analytics 13.0 in case of a server hardware failure. Three components are required to be backed up:

• PBSA Web Application and Parser

• PBSA Data Collector

21.1 Backing up PBSA

It is highly recommended to backup PBSA onto a backup machine or to disk so that if the pro-duction machine or disk fails it does not affect the backup.


Chapter 21 Disaster Recovery

21.1.1 Backing up the Web Application and Parser

Perform these steps on the machine where the PBSA Web Application and Parser have been deployed:

1. Verify that parsing of the PBS Professional accounting logs is not currently in prog-ress - see section 8.8, “Parsing the accounting logs”, on page 110).

2. As the root user, stop the PBSA service - see Chapter 14, "PBSA service commands", on page 141.

3. On the backup machine, create a backup directory structure as shown in Figure 21-1.

Figure 21-1:.Backup directory structure


Disaster Recovery Chapter 21

4. Login to the server where the Web Application and Parser has been deployed using the login credentials of the PBSA service user (specified during the installation).

5. Zip the INSTALL_DIR_13.0/portal directory and copy it to the PBSA-backup-YYYY-MM-DD/analytics-engine/portal directory on the backup machine/disk. Make sure there are no errors or warnings at the time of the zipping operation.

6. If PBSA is being backed up to an Operating System which is different from the one on which it is installed, perform the database dump manually, and copy the database dump file to the PBSA-backup-YYYY-MM-DD/analytics-engine/dbdump directory - see section 21.3, “Backing up the PBSA database”, on page 329.

7. Copy /etc/init.d/pbsworks-pbsa to the PBSA-backup-YYYY-MM-DD/analytics-engine/init.d directory.

8. Copy /etc/pbsworks-pbsa.conf to the PBSA-backup-YYYY-MM-DD/analytics-engine/conf directory.

21.1.2 Backing up the Data Collectors

Perform these steps on all machines where the PBSA Data Collector has been deployed:

1. As the root user, stop all PBSA Data Collectors - see Chapter 14, "PBSA service commands", on page 141.

2. On the backup machine, create a backup directory structure as shown in Figure 21-1 for every Data Collector installed. Create a unique directory called PBSA-backup-YYYY-MM-DD/analytics-datacollector-<DC_HOSTNAME>.

3. Login to the machine where the Data Collector has been deployed using the login credentials of the PBSA service user (specified during the installation of PBSA).

4. Zip the INSTALL_DIR_13.0/dc directory and copy it to the PBSA-backup-YYYY-MM-DD/analytics-datacollector-<DC_HOSTNAME>/dc direc-tory on the backup machine/disk. Make sure there are no errors or warnings at the time of the zipping operation.

5. Copy /etc/init.d/pbsworks-dc to the PBSA-backup-YYYY-MM-DD/analytics-datacollector-<DC_HOSTNAME>/init.d directory.

6. Copy /etc/pbsworks-dc.conf to the PBSA-backup-YYYY-MM-DD/ana-lytics-datacollector-<DC_HOSTNAME>/conf directory.

Now that all components have been backed up, start the PBSA and Data Collector services as the root user.



21.2 Restoring PBSA

21.2.1 Restoring the Web Application and Parser

1. Verify that the previous installation of PBSA is completely removed. - see Chapter 7, "Uninstalling PBSA", on page 99.

2. Use the latest backup of PBSA for the restoration. The latest backup is determined by the backup date in the folder name.

3. Login to the server where the Web Application and Parser has been deployed using the login credentials of the PBSA service user (specified during the installation).

4. Unzip the portal directory from the PBSA-backup-YYYY-MM-DD/analyt-ics-engine/portal directory into the INSTALL_DIR_13.0/portal direc-tory. Make sure there are no errors or warnings at the time of the unzipping operation.

5. If PBSA is being backed up to an Operating System which is different from the one on which it is installed, use the database dump file from the PBSA-backup-YYYY-MM-DD/analytics-engine/dbdump directory to restore the database - seesection 21.4, “Restoring the PBSA database”, on page 330.

6. Copy PBSA-backup-YYYY-MM-DD/analytics-engine/init.d/pbsworks-pbsa to /etc/init.d/pbsworks-pbsa.

7. Copy PBSA-backup-YYYY-MM-DD/analytics-engine/conf/pbsworks-pbsa.conf to /etc/pbsworks-pbsa.conf.

21.2.2 Restoring the Data Collectors

Perform these steps on all machines where the PBSA Data Collector has been deployed:

1. Unzip the dc directory from the PBSA-backup-YYYY-MM-DD/analytics-datacollector-<DC_HOSTNAME>/dc directory into the INSTALL_DIR_13.0/dc directory. . Make sure there are no errors or warnings at the time of the unzipping operation.

2. Copy PBSA-backup-YYYY-MM-DD/analytics-datacollector-<DC_HOSTNAME>/init.d/pbsworks-dc to /etc/init.d/pbsworks-dc.

3. Copy PBSA-backup-YYYY-MM-DD/analytics-datacollector-<DC_HOSTNAME>/conf/pbsworks-dc.conf to /etc/pbsworks-dc.conf.



21.3 Backing up the PBSA database

It is highly recommended to backup the PBSA database to a backup machine or to disk so that if the production machine or disk fails it does not affect the backup.


1. Verify that parsing of the PBS Professional accounting logs is not currently in prog-ress - see section 8.8, “Parsing the accounting logs”, on page 110.

2. As the root user, stop the PBSA and Data Collector services - see Chapter 14, "PBSA service commands", on page 141.

3. Run the following command:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:INSTALL_DIR_13.0/portal/thirdparty/monetdb/lib/

4. Navigate to the following directory by running the command:

cd INSTALL_DIR_13.0/portal/thirdparty/monetdb/scripts

5. Start the database service by running the command:

./startmonetdb.sh INSTALL_DIR_13.0/portal/thirdparty/monetdb pbsworks-data


cd INSTALL_DIR_13.0/portal/thirdparty/monetdb/bin

7. Run the command:

./mclient -p 9200 -d pbsworksdb -D > <BACKUP_LOCATION>/pbsworksdb.sql

8. You will be prompted to enter a user id and password. Enter the database super user “pbsworks” and the super user password.



10. Stop the database service by running the command:

./stopmonetdb.sh INSTALL_DIR_13.0/portal/thirdparty/monetdb pbsworks-data

11. As the root user, start the PBSA and Data Collector services - see Chapter 14, "PBSA service commands", on page 141.



21.4 Restoring the PBSA database


1. Verify that parsing of the PBS Professional accounting logs is not currently in prog-ress - see section 8.8, “Parsing the accounting logs”, on page 110.

2. As the root user, stop the PBSA and Data Collector services - see Chapter 14, "PBSA service commands", on page 141.

3. If you are restoring from a different operating system, perform the following steps.

a. Unzip the PBSA installer binary to a temporary directory:

unzip PBSA_13.0_Linux_x86_64.bin -d /tmp/pbsa_installer

b. Navigate to /tmp/pba_installer/$IA_PROJECT_DIR$/repo/mon-etdb_linux64.

c. Unzip the monetdb_64.zip file to a temporary directory:

unzip monetdb_64.zip -d /tmp/monetdb_64

d. If you are restoring the database to a CentOS platform, then:

- copy the /tmp/monetdb_64/bin directory to INSTALL_DIR_13.0/portal/thirdparty/monetdb.

- copy the /tmp/monetdb_64/lib directory to INSTALL_DIR_13.0/portal/thirdparty/monetdb.

e. If you are restoring the database to a SUSE platform, then:

- copy the /tmp/monetdb_64/bin_SuSE10 directory to INSTALL_DIR_13.0/portal/thirdparty/monetdb.

- copy the /tmp/monetdb_64/lib_SuSE10 directory to INSTALL_DIR_13.0/portal/thirdparty/monetdb.

- Remove the directory INSTALL_DIR_13.0/portal/thirdparty/monetdb/bin

- Remove the directory INSTALL_DIR_13.0/portal/thirdparty/monetdb/lib.

- Rename the directory INSTALL_DIR_13.0/portal/thirdparty/monetdb/bin_SuSE10 to INSTALL_DIR_13.0/portal/third-party/monetdb/bin.

- Rename the directory INSTALL_DIR_13.0/portal/thirdparty/monetdb/lib_SuSE10 to INSTALL_DIR_13.0/portal/third-



party/monetdb/lib.


export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:INSTALL_DIR_13.0/portal/thirdparty/monetdb/lib/

5. Navigate to the following directory:


6. Start the database service:

./startmonetdb.sh INSTALL_DIR_13.0/portal/thirdparty/monetdb pbsworks-data

7. Navigate to the following:

cd INSTALL_DIR_13.0/portal/thirdparty/monetdb/bin


./mclient -p 9200 -d pbsworksdb < <BACKUP_LOCATION>/pbsworksdb.sql

9. You will be prompted to enter a user id and password. Enter the database super user “pbsworks” and the super user password.

10. Navigate to the following directory by issuing the following command:


11. Stop the database service by issuing the following command:

./stopmonetdb.sh INSTALL_DIR_13.0/portal/thirdparty/monetdb pbsworks-data

12. As the root user, start the PBSA and Data Collector services - see Chapter 14, "PBSA service commands", on page 141.


Chapter 22

Peer SchedulingPeer scheduling allows separate PBS complexes to automatically run jobs from each other’s queues. This means that you can dynamically balance the workload across multiple, separate PBS Professional complexes. These cooperating complexes are referred to as “Peers”. When Complex A pulls a job from Complex B, Complex A is the “pulling” complex and Complex B is the “furnishing” complex.

The diagram belows shows how PBSA will need to be deployed for sites where peer schedul-ing and failover are enabled. As you can see, a PBSA Data Collector must be installed on each primary server.


Chapter 22 Peer Scheduling

In addition, the pbsa-config-nodes command has changed to support peer scheduling. When configuring node details for a PBS Professional complex you must now provide a list of furnishing servers.

Using the diagram above, to configure the node details for the Bangalore PBS Professional complex, you would issue the following command:

pbsa-config-nodes Bangalore bangalore.ms.com pbsnodes.out --furnishing-servers=Troy,MV

You can use the pbsa-setup-counsellor command to obtain a list of furnishing serv-ers for a PBS Professional complex.


Chapter 23

PBSA HooksPBSA reports upon various common PBS Professional resources such as CPUs, memory, and walltime. In addition, PBSA can report upon site-specific resources known as custom resources. This type of reporting requires a PBSA Hook - a mechanism for adding a custom resource to PBSA. Once a custom resource is configured, it can be added to a chart.

The following steps are necessary to create a PBSA Hook to add a custom resource to PBSA:

1. Define the custom resource to PBSA by updating the custom resource configuration file.

2. Update the node details configuration file with information about the custom resource.

3. If PBS Professional accounting logs have already been parsed and loaded into PBSA, then discard the parsed data and reparse all accounting logs. For more information see Chapter 20, "Reparsing the accounting logs", on page 323.

When special processing of a custom resource is required, you can create a converter class to perform this specialized conversion.

For example, a site charges a client for the use of a cluster based on the GPU walltime used per job. The client is charged $10 for each GPU hour. A converter class can be written to calculate this charge.

For more information about creating converter classes see section 23.3, “Writing a converter class”, on page 344.


Chapter 23 PBSA Hooks

Below are examples of informational messages that are helpful for verifying that the custom resource and associated converter class are parsed and loaded successfully. These messages are written to the parser log file:

Table 23-1: Successful parsing and processing messages

Message Description

Going to read custom resource configuration Beginning to process custom resources (PBSA Hooks).

You have provided a custom hook class for this resource, going to verify class name : <classname>

Verifying that the class denoted by <classname> that is associated with the custom resource exists.

Successfully able to load hook class Custom resource class is found and loaded.

Custom Resource processing finished Processing of custom resources is complete.

*****No custom resource configured ***** There were no custom resources defined.


PBSA Hooks Chapter 23

23.1 Define the custom resource to PBSA

The custom resource must be defined to PBSA by adding it to the following XML file:

INSTALL_DIR_13.0/portal/binaries/parser/hooks/customresources.xml

The following XML tags define a custom resource to PBSA:

A <resource> must be added to this XML file for each custom resource.

The format and location of the custom resource file has changed. This file is now an XML file rather than a tab formatted text file.

<pbsacustomresource> <resources> <resource> <attributename> </attributename> <datatype> </datatype> <displayname> </displayname> <displaytype> </displaytype> <calculatorclass> </calculatorclass> <unittype> </unittype> </resource> </resources></pbsacustomresource>




T

finition

Default Optional

a None No

d None No

d If this element is not defined, PBSA will display the name of the custom resource as defined by the XML tag <attributename>, replacing any periods ( . ) with an underscore ( _ ).

For example: Resource_List.ngpus will be displayed as Resource_List_ngpus.

Yes

38 PB

he following table describes the custom resource definition XML elements:

Table 23-2: XML Elements of the Custom Resource De

XML Element

Description Valid Values

ttributename Name of the custom resource as specified in the PBS Pro-fessional accounting logs.

Must be a valid PBS Professional custom resource.

Ex. Resource_List.ngpus

atatype The data type of the custom resource.

Integer

String

Numeric

isplayname Name of the custom resource as it will be represented in the user interface.

Alphanumeric ( A-Z ), ( a-z ), ( 0-9 )

underscore ( _ )

space

Ex. NGPUS or N_GPUS or No GPUS

PB 339


di If this element is not defined then the cus-tom resource will be available as both a Dimension and a Mea-sure.

Yes.

ca None Yes

un pes

ypes

NONE Yes.

finition

Default Optional


splaytype Use this tag to specify if the custom resource will be exposed as a Measure or a Dimension.

dimension

measure

lculatorclass Specify the fully qualified calculator class which will be used for converting the value of a custom resource prior to populating the PBSA database. For information about writing a converter class see section 23.3, “Writ-ing a converter class”, on page 344.

None

Ex. com.altair.pbsworks.customre-source.CustomNodeValue

ittype Unit of measurement. STRING

TIME

MEMORY

NONE

String/character data ty

Time data types

Memory data types

Numeric/integer data t

Table 23-2: XML Elements of the Custom Resource De

XML Element

Description Valid Values


The following example defines a custom resource to PBSA called “Resource_List.ngpus” . The custom resource has the following attributes:

• The custom resource is a data type of integer.

The following defaults apply:

• The custom resource is available as both a Dimension and a Measure.

• The display name of this custom resource defaults to “Resource_List_ngpus”.

• The value of this custom resource is stored as a numeric value within PBSA.

<pbsacustomresource>

<resources>

<resource>

<attributename>Resource_List.ngpus</attributename>

<datatype>Integer</datatype>

</resource>

</resources>

</pbsacustomresource>



The following example defines a custom resource to PBSA called “Resource_List.cm”. The custom resource has the following attributes:

• The custom resource is a data type of string.

• The display name is “cm-measure”.

• The custom resource is available as a Measure within PBSA.




<resources>

<resource>

<attributename>ResourceList.cm</attributename>

<datatype>String</datatype>

<displayname>cm-measure</displayname>

<displaytype>measure</displaytype>

</resource>

</resources>




The following example defines a custom resource to PBSA called “Resource_List.dm”. The custom resource has the following attributes:

• The custom resource is a data type of string.

• The display name is “DM”.

• The custom resource is available as a Dimension within PBSA.




<resources>

<resource>

<attributename>ResourceList.dm</attributename>

<datatype>String</datatype>

<displayname>DM</displayname>

<displaytype>dimension</displaytype>

</resource>

</resources>



PB 343


23

In file must be updated with information ab ils configuration file see section 18.5.6, “D defined to PBSA, then the custom res

IN - - - - 2-01-01 2012-12-31 - - -- 2-01-01 2012-12-31 - - -- - - - custom -- - - - custom -- - - - custom -- - - - custom -- ] - - - custom - - GB - - - custom - - - - - custom -- -01-01 2012-12-31 - - -- - - - custom -- - - - custom -- - - - custom -- - - - custom -- ] - - - custom - - GB - - - custom - - - - - custom -


.2 Updating the node details configuration file

order to support a new custom resource, each node and virtual node in the node details configuration out the custom resource. For more information about adding a custom resource to the node detaefining resources”, on page 290. For example, if the custom resource “Resource_List.ngpus” isource must be added to the node details file:

STALL_SNAPSHOT RANGE blrhpc3 - - - - - - - - - - - blrhpc3 hp.prog.com - - - - - - - 201 - blrhpc3 - blrhpc3n1 blrhpc3n1v[*] 0-2 - - - - 201 - - - - - - string STATIC resources_available.arch linux - - - - - - string STATIC resources_available.host blrhpc3n1 - - - - - - string STATIC resources_available.mem 32900260 - - - - - - string STATIC resources_available.npus 12 - - - - - - string STATIC resources_available.vnode blrhpc3nv[* - - - - - - string STATIC resources_available.nodeclass 1CORE4 - - - - - - string STATIC resources_available.ngpus 3 - blrhpc3 - blrhpc3n1 blrhpc3n1v[9] - - - - - 2012 - - - - - - string STATIC resources_available.arch linux - - - - - - string STATIC resources_available.host blrhpc3n1 - - - - - - string STATIC resources_available.mem 32900260 - - - - - - string STATIC resources_available.npus 12 - - - - - - string STATIC resources_available.vnode blrhpc3nv[9 - - - - - - string STATIC resources_available.nodeclass 1CORE4 - - - - - - string STATIC resources_available.ngpus 3


23.3 Writing a converter class

You can write your own converter class to process a PBS Professional custom resource before the data is loaded into the PBSA database.

In the following example, the converter class will calculate a custom value for a node by dividing the resources_used.ncpus by 12.

Follow these steps to create a converter class after installing PBSA.

1. Download Eclipse from http://www.eclipse.org/downloads/packages/eclipse-stan-dard-431/keplersr1.

2. Install the downloaded binary and create a new workspace by launching Eclipse.

3. Add JAVA to the workspace (JRE 6 is required)

a. Click on Window > Preferences.

b. Expand the Java category.

c. Click the Installed JRE’s option.

d. Click Add.

e. Click the STANDARD VM option and click Next.


http://www.eclipse.org/downloads/packages/eclipse-standard-431/keplersr1

http://www.eclipse.org/downloads/packages/eclipse-standard-431/keplersr1


f. Set the JRE home directory to INSTALL_DIR_13.0/portal/third-party/java/jre and click Finish.



4. Create a Java project.

a. Click File > New > Project....

b. Click the Java Project option and click Next.



c. Enter the Project name.

d. Specify the JRE added in Step 3.

e. Click Finish.



5. Add the PBSA jar files to the project’s build path.

a. Right-click the Java project that you just created.

b. Click Build Path > Configure Build Path.

c. Click Add External Jars.

d. Browse to INSTALL_DIR_13.0/portal/binaries/framework/webapps/parser/WEB-INF/lib and select all libraries.

e. Click OK.



6. Create the custom resource converter class.

a. Click File > New > Class.

b. Specify the name of the class and its package.



7. Enter the code for the converter class.

A custom resource converter class has to implement the IDerivedAttribute interface and override the deriveValue (Map coreValueMap, String attribute) method.

The deriveValue method calculates and returns the value of a custom resource.

For more information about the converter class API see section 23.3.1, “Converter class API”, on page 354.

In the following example, the converter class will calculate a custom value for a node by dividing the resources_used.ncpus by 12.



8. Create a jar file for the converter class.

a. Right click the project.

b. Click Export.

c. Enter “jar” in the Select an export destination field.

d. Click the Jar file option.

e. Click Next.



f. In the Select the export destination field, enter the location where the jar file will be exported.

g. Click Finish.

h. Copy the exported jar file to INSTALL_DIR_13.0/portal/binaries/parser/lib.


PB 353


On le with information about the custom res file see section 18.5.6, “Defining res

Th e information:

Fin

IN - - - - 2-01-01 2012-12-31 - - -- 2-01-01 2012-12-31 - - -- - - - custom -- - - - custom -- - - - custom -- - - - custom -- ] - - - custom - - GB - - - custom - - - - - custom -

< calculatorclass> <


ce a converter class is created, update each node and virtual node in the node details configuration fiource. For more information about adding a custom resource to the node details configuration ources”, on page 290.

e following is an example of the node details configuration file updated with the custom resourc

ally, update the customresources.xml file with the new custom resource information:

STALL_SNAPSHOT RANGE blrhpc3 - - - - - - - - - - - blrhpc3 hp.prog.com - - - - - - - 201 - blrhpc3 - blrhpc3n1 blrhpc3n1v[*] 0-2 - - - - 201 - - - - - - string STATIC resources_available.arch linux - - - - - - string STATIC resources_available.host blrhpc3n1 - - - - - - string STATIC resources_available.mem 32900260 - - - - - - string STATIC resources_available.npus 12 - - - - - - string STATIC resources_available.vnode blrhpc3nv[* - - - - - - string STATIC resources_available.nodeclass 1CORE4 - - - - - - string STATIC resources_available.nodevalue 3

pbsacustomresource> <resources> <resource> <attributename>Resource_List.nodevalue</attributename> <datatype>Numeric</datatype> <displayname>Custom Node Value</displayname> <displaytype>measure</displaytype> <calculatorclass>com.altair.pbsworks.customresource.CustomNodeValue</ </resource> </resources>pbsacustomresource>


23.3.1 Converter class API

A custom resource converter class has to implement the IDerivedAttribute interface and over-ride the deriveValue (Map coreValueMap, String attribute) method.

Override the method deriveValue with custom code that will calculate and return the value of a custom resource.

The parameter attribute is the name of the custom resource as defined in the customre-sources.xml file. In our example, this would be the custom resource “Resource_List.nodevalue”.

The parameter coreValueMap is a map containing job information. A map is an object that maps keys to values. Some of this information is obtained from the PBS Professional account-ing logs, and some of it is extrapolated from information contained in the PBS Professional accounting logs.

PBSA also stores information about custom resources related to a job in this map. For exam-ple, if a custom resource called “Resource_List.nodevalue” was added to the customre-sources.xml file, then this resource and its value will also be contained in the map.

The information contained in this map, can be used by your own converter class to calculate the value of a custom resource.

The following table describes the default keys (job information), along with their values, that are stored in the map.

Table 23-3: Default Converter Class Map

Map Key Description

queue The name of the queue in which the job executed.

group The group name under which the job executed.

user The user name under which the job executed.

exec_vnode A list of each vnode with vnode-level, consumable resources from that vnode.

exec_host A list of each vnode with vnode-level, consumable resources allocated from that vnode.

Exit_status The exit status of the job.

account The account name associated with the job.

project The job’s project name.

customgroup The custom group assigned to the job.



Resource_List.walltime The walltime requested by the job.

Resource_List.place The placement requested by the job.

Resource_List.software The software requested by the job.

Resource_List.mem The memory requested by the job.

resources_used.mem The memory used by the job.

Resource_List.arch The architecture requested by the job.

Resource_List.cput The amount of CPU time requested by the job.

resources_used.cput The amount of CPU time used by the job.

etime The time when the job became eligible to run, i.e. was enqueued in an execution queue and was in the “Q” state.

ctime The time when the job was created (first submitted).

qtime The time when the job was queued into current queue.

start The time when job execution started.

end The time when this accounting record was written.

jobId The job identifier.

Resource_list.ncpus The number of CPUs requested by the job.

resources_used.ncpus The number of CPUs used by the job.

walltime_split The walltime used by the job. It is calculated by subtracting the start time from the end time.

resourced_used.walltime The actual walltime reported by PBS Professional.

recordType The state of the job.

Resource_List.vmem The virtual memory requested by the job.

resources_used.vmem The virtual memory used by the job.

Resource_List.nodeclass The nodeclass requested by the job.

Table 23-3: Default Converter Class Map

Map Key Description



23.4 Troubleshooting PBSA Hooks

This section provides guidelines on how to troubleshoot common problems with PBSA Hooks. Errors found while processing PBSA Hooks will be written to:

INSTALL_DIR_13.0/portal/logs/parser.log

23.4.1 Mandatory elements missing for a custom resource

I added a new custom resource. Once parsing began, I saw the following warning message in the parser log file:

Two XML element are mandatory when defining a custom resource to PBSA:

• attributename - The name of the custom resource as specified in the PBS Professional accounting logs.

• datatype - The data type of the custom resource.

Either one or both of these elements are missing from the custom resource configuration file. PBSA will ignore this custom resource and continue with the parsing process.

To correct the problem, add the missing XML elements to the custom resource configuration file, discard any parsed data, and reparse all PBS Professional accounting logs.

23.4.2 Added a custom resource but did not reset data

I added a new custom resource. Once parsing began, I saw the following error message in the parser log file:

If changes are made to PBSA Hooks after the initial parsing cycle is complete, any parsed data must be discarded and all PBS Professional accounting logs have to be reparsed. For more information on how to perform a data reset see Chapter 20, "Reparsing the accounting logs", on page 323.

<attr> and <datatype> are mandatory elements while configuring new custom resource, hence skipping this resource

Custom resource file updated, please do a data reset everytime customresources.xml file is updated.



23.4.3 Problem loading PBSA Hook converter class

I have created a custom resource converter class. Once parsing began, I saw the following error message in the parser log file:

PBSA could not load the Java class associated with a custom resource. Verify that the jar file associated with the converter class was copied to the following location:

INSTALL_DIR_13.0/portal/binaries/parser/lib

If the jar file has been copied to the correct location and this message persists, then contact systems support.

For more information on how to create a custom resource converter class see section 23.3, “Writing a converter class”, on page 344.

23.4.4 Checksum validation for custom resource file

Once parsing began, I saw the following error message in the parser log file:

There is a data integrity issue with the custom resource configuration file. Contact systems support.

23.4.5 Error configuring custom resource


There is a problem processing a custom resource. Additional information is logged to help debug the problem. Correct the issue, discard any parsed data, and reparse all PBS Profes-sional accounting logs. If you are unable to debug and correct the issue, contact systems sup-port.

error in loading custom hook class

Error while validation checksum for customresources.xml file.

Error while configuring custom resource



23.4.6 Skipping configuration of custom resource hook


There was a problem with configuring a PBASA hook for the custom resource denoted by <resource>. PBSA will ignore this custom resource and continue with the parsing process. Correct the issue with the custom resource, discard any parsed data, and reparse all PBS Pro-fessional accounting logs. If you are unable to debug and correct the issue, contact systems support.

Skipping configuration of hook for resource <resource>


Chapter 24

Using the PBSA Migration ToolWARNING: Please ensure that you run the migration tool before installing PBSA 13.0. If the migration tool is ran after installing PBSA 13.0, then there is high probability that charts, dashboards, and slideshows saved with PBSA 13.0 will be overwritten during the migration.

Follow these steps to migrate the 12.3.x or 12.4.x user charts, dashboards, and slideshows to a 13.0 format. These steps must be executed as root user.

1. Download the migration tool from the PBSA User Area. The migration tool is con-tained in the zip file: migration.zip.

2. Unzip the migration.zip file and save the files stored in the zip file to the instal-lation directory of PBSA 12.3.x or 12.4.x. The location of the unzipped files will be denoted by UNZIP_LOCATION.

3. Change the permissions of the migration scripts to make them executable by issuing the following command:

chmod +x UNZIP_LOCATION/migration/bin/*

4. Navigate to the migration script directory by issuing the following command:

cd UNZIP_LOCATION/migration/bin

5. Run the migration script:

./export-13.0-format.sh

6. Stop the 12.3.x or 12.4.x PBSA service and Data Collector services. For more infor-mation see section 14.1, “PBSA service commands”, on page 141.

7. It is advisable to keep the backup of the existing PBSA 12.3.x or 12.4.x installation until all charts, dashboards, and slideshows are successfully migrated to a PBSA


Chapter 24 Using the PBSA Migration Tool

13.0 format. So delete the following registry file created by the PBSA installation process:

rm –f /var/.com.zerog.registry.xml

8. Install PBSA 13.0 in a different installation location than PBSA 12.3.x or 12.4.x. Do not start the PBSA 13.0 service.

9. Navigate to the migration script directory by issuing the following command:

cd UNZIP_LOCATION/migration/bin

10. Run the migration script:

./import-13.0-format.sh


Glossary of TermsThis chapter describes the terms used in PBS Analytics documentation.

Account An account is an arbitrary character string, which may have meaning to one or more hosts in the batch system. Frequently, an account is used as a grouping for charging for the use of resources.

Accounting Log The PBS Professional accounting log is a text file containing information about a clus-ter’s HPC job activity.

Administrator, PBSA AdministratorA person who administers PBSA, performing functions such as downloading, installing, upgrading, configuring, or managing PBSA. A PBSA administrator must have an account with root privilege. Administrator is distinguished from “site administrator”, although often these are the same person.

ChartA charts is a visual representation of data. Charts are used to ease understanding of large quantities of data and the relationships between parts of the data.

ClusterA relatively homogeneous set of systems that are used as if they are a single machine.

CommandsPBSA supplies commands to assist with the configuration of PBSA.


Glossary of Terms

ComplexA PBS complex consists of the machines running one primary server+Scheduler (plus, optionally, a secondary backup server+Scheduler) and all the machines on which the MoMs (attached to this server+Scheduler) are running. A complex can be a heteroge-neous mix of system architectures, and can include one or more clusters.

ConfigurationPost-installation administrative activities that supply site specific information to PBSA.

Consumable resourceA consumable resource is a resource that is reduced or taken up by being used. Examples of consumable resources are memory or CPUs.

CPUHas two meanings, one from a hardware viewpoint, and one from a software viewpoint:

1. A core. The part of a processor that carries out computational tasks. Some sys-tems present virtual cores, for example in hyperthreading.

2. Resource required to execute a program thread. PBS schedules jobs according, in part, to the number of threads, giving each thread a core on which to execute. The resource used by PBS to track CPUs is called “ncpus”. The number of CPUs available for use defaults to the number of cores reported by the OS. When a job requests one CPU, it is requesting one core on which to run.

Custom groupCustom groups are specific to PBS Analytics, and are an alternative for tracking usage of HPC resources at a project level for sites running versions of PBS Professional prior to 11.2.

Custom resourceA resource that is not defined in PBS as shipped. Custom resources are created by the PBS administrator or by PBS for some systems.

Daily ParsingA PBSA activity where information from the PBS Professional accounting logs is parsed, and then loaded into a database. Once the PBS Professional accounting logs have been parsed for the first time (see "Initial Parsing"), daily parsing is scheduled by default to occur at 1:00 am, and will parse any newly generated accounting logs.

DashboardA dashboard is a visual display of the most important information needed to achieve one or more objectives; consolidated and arranged on a single screen so the information can be monitored at a glance.


Glossary of Terms

DimensionA field or variable that is displayed on a PBSA chart. Dimensions are the independent variable (often denoted by the x-axis) whose variation does not depend on another vari-able. Examples are users, software, servers, and nodes. PBSA requires that Dimensions be displayed on the x-axis or z-axis (in the case of a three dimensional chart).

Demonstration DatabaseA database that is loaded with demonstration data. An installation option is available that installs PBSA with a demonstration database, so that all out-of-the box charts, dash-boards, and slideshows can be viewed, and all key features of PBSA can be accessed and evaluated.

Execution hostA computer which runs PBS jobs. An execution host is a system with a single operating system (OS) image, a unified virtual memory space, one or more CPUs and one or more IP addresses. Systems like Linux clusters, which contain separate computational units each with their own OS, are collections of hosts. Systems such as the SGI ICE are also collections of hosts. The SGI Altix 4700 is a single execution host.

An execution host can be comprised of one or more vnodes. For example, the SGI Altix 4700, while being a single execution host, can contain multiple vnodes, where each vnode is a blade. On the SGI ICE, each blade is treated as a vnode. See "Vnode".

Finished jobsJobs whose execution is done, for any reason:

• Jobs which finished execution successfully and exited

• Jobs terminated by PBS while running

• Jobs whose execution failed because of system or network failure

• Jobs which were deleted before they could start execution

Furnishing ServerThe server from which the job is pulled when peer scheduling is implemented.

GroupA collection of system users. A user must be a member of at least one group, and can be a member of more than one group.

Head NodeMachine where the PBS Server is installed.

High Performance Computing, HPCHigh Performance Computing (HPC) most generally refers to the practice of aggregating computing power in a way that delivers much higher performance than one could get out


Glossary of Terms

of a typical desktop computer or workstation in order to solve large problems in science, engineering, or business.

HolidayA non-working day usually associated with a special day of celebration.

HostA machine running an operating system. A host can be made up of one or more vnodes.

Initial ParsingA PBSA activity where information from the PBS Professional accounting logs is parsed, and then loaded into a database. Initial parsing is when the accounting logs are parsed for the first time after installation and configuration of PBSA is complete. Initial parsing may take anywhere from several hours to several days depending upon the amount of data stored in the PBS Professional accounting log(s), number of processors, and amount of available RAM.

Job or Batch jobA unit of work managed by PBS. A job is a related set of tasks, created and submitted by the user. The user specifies the resources required by the job, and the processes that make up the job. When the user submits a job to PBS, the user is handing off these tasks to PBS to manage. PBS then schedules the job to be run, and manages the running of the job, treating the tasks as parts of a whole. A job is usually composed of a set of directives and a shell script.

Job ID, Job identifierWhen a job is successfully submitted to PBS, PBS returns a unique identifier for the job. Format:

sequence_number[.server][@new server]

Job stateA job exists in one of the possible states throughout its existence within the PBS system. For example, a job can be queued, running, or exiting.

License server A centralized computer software system which provides access tokens, or keys, to client computers in order to enable licensed software to run on them.

Measures A field or variable that is displayed on a PBSA chart. Measures are the dependent vari-able (often denoted by the y-axis) whose value changes dependent on the other variables. Examples are total numbers of jobs, average memory used, and total CPU walltime. Measures are displayed on the y-axis. The PBSA Web Application refers to Measures as Values.


Glossary of Terms

MoMThe daemon which runs on an execution host, managing the jobs on that host. MoM is the informal name for the process called pbs_mom. One MoM runs on each execution host.

MoM runs each job when it receives a copy of the job from the server. MoM creates a new session that is as identical to the user’s login session as possible. For example under UNIX, if the user’s login shell is csh, then MoM creates a session in which .login is run as well as .cshrc. MoM returns the job’s output to the user when directed to do so by the server.

MoM is a reverse-engineered acronym that stands for “Machine Oriented Mini-server”.

Mother SuperiorMother Superior is the MoM on the head or first host of a multihost job. Mother Superior controls the job, communicates with the server, and controls and consolidates resource usage information. When a job is to run on more than one execution host, the job is sent to the MoM on the primary execution host, which then starts the job.

NodeNo longer used. See "Execution host".

NodeclassA custom resource that groups or classifies PBS Professional nodes, PBSA automatically assign a nodeclass based on cores and memory when node information is added to PBSA via the pbsa-config-nodes command.

Node DetailsA map of your site’s PBS Professional cluster, that is each PBS Professional server and its associated execution hosts. This information is stored in a configuration file, and is used by PBSA to provide HPC resource usage statistics.

Non-consumable resourceA non-consumable resource is a resource that is not reduced or taken up by being used. Examples of non-consumable resources are Boolean resources and walltime.

Non-Prime Time HoursNon-working hours.

Owner, Job ownerThe user who submitted a specific job to PBS.

PBS Analytics, PBSAA job accounting and reporting solution that provides PBS Works administrators with advanced analytics to support data-driven planning and decision making. Data from mul-tiple PBS Professional servers can be consolidated, providing a global view of your HPC usage for chargeback, capacity planning, troubleshooting and project management.


Glossary of Terms

PBSA Data Collector, Data CollectorA software component of PBSA, installed on execution hosts, responsible for making the PBS Professional accounting logs accessible to the PBSA engine.

PBSA Engine, PBSA Web ApplicationA software component of PBSA that consists of web services, a database, and an analyt-ics engine responsible for the analysis and visualization of HPC resources.

PBS_HOMEThe path containing PBS files. The path under which PBS files are installed on the local system.

PBS_EXECThe path containing PBS executables. The path under which PBS executables are installed on the local system.

PBSA HookA mechanism for adding a custom resource to PBSA.

PBSA Parser, ParserA software component of PBSA that is responsible for parsing the PBS Professional accounting logs and loading this information into the PBSA database.

PBSA Service UserAn existing user on the server where PBSA is being installed. It will be used to register the PBSA service. The PBSA Web Application and database services will be started under this user.

PBS WorksPBS Works is a suite of products for comprehensive, secure workload management for high-performance computing (HPC) environments.

PBS ProfessionalA workload management system consisting of a server, a Scheduler, and any number of execution hosts each managed by a MoM. PBS accepts batch jobs from users, and sched-ules them on execution hosts according to the policy chosen by the site. PBS manages the jobs and their output according to site-specified policy.

Peer schedulingA feature allowing different PBS complexes to automatically run each others’ jobs. This way jobs can be dynamically load-balanced across the complexes. Each complex involved in peer scheduling is called a peer.

Prime Time HoursWorking hours.


Glossary of Terms

ProjectIn PBS, a project is a way to group jobs independently of users and groups. A project is a tag that identifies a set of jobs. Each job’s project attribute specifies the job’s project.

QueueA queue is a named container for jobs at a server. There are two types of queues in PBS: routing queues and execution queues. A routing queue is a queue used to move jobs to other queues including those that exist on other PBS servers. A job must reside in an exe-cution queue to be eligible to run and remains in an execution queue during the time it is running. In spite of the name, jobs in a queue need not be processed in queue order (first-come first-served or FIFO).

ResourceA resource can be something used by a job, such as CPUs, memory, high-speed switches, scratch space, licenses, or time, or it can be an arbitrary item defined for another purpose. PBS has built-in resources, and allows custom-defined resources.

SchedulerThe PBS scheduler is the daemon which implements the site’s job scheduling policy con-trolling when and where each job is run. The scheduler is the process called pbs_sched.

SchedulingThe process of selecting which jobs to run when and where, according to a predetermined policy. Sites balance competing needs and goals on the system(s) to maximize efficient use of resources (both computer time and people time).

ServerThe central PBS daemon, which does the following:

• Handles PBS commands

• Receives and creates batch jobs

• Sends jobs for execution

The server is the process called pbs_server.

Each PBS complex has one primary server, and if the complex is configured for failover, a secondary server.

The server contains a licensing client which communicates with the licensing server for licensing PBS jobs.


Glossary of Terms

UserHas two meanings:

3. A person who submits jobs to PBS, as differentiated from Operators, Managers and administrators.

4. A system user, identified by a unique character string (the user name) and by a unique number (the user ID). Any person using the system has a username and user ID.

User ID, UIDA unique numeric identifier assigned to each user.

ValuesThe PBSA Web Application refers to Measures as Values. Values are the dependent vari-able (often denoted by the y-axis) whose value changes dependent on the other variables. Examples are total numbers of jobs, average memory used, and total CPU walltime. Val-ues are displayed on the y-axis.

VnodeA virtual node, or vnode, is an abstract object representing a set of resources which form a usable part of an execution host. This could be an entire host, or a nodeboard or a blade. A single host can be made up of multiple vnodes. Each vnode can be managed and scheduled independently. Each vnode in a complex must have a unique name. Vnodes can share resources, such as node-locked licenses.


IndexAaccount 361accounting

account 361adding a PBS Complex 139administrator 361adminSettings.xml 308, 309, 312Advanced Configuration 258, 259, 264, 267, 274, 275, 301

custom group configuration file 267custom resource configuration file 274exit status configuration file 264holiday configuration file 258, 259, 301node detail configuration file 275

advanced configuration 109, 124application definitions 175application metrics 15

Bbatch job 364

Cchange database password 206, 207Changing the accounting log location 313Changing the license schema 316Changing web application hostname

port 314

chart 361chart rendering 315classifying PBS Professional nodes 20cluster 361COMMAND_HOME 105commands 361complex 362Configuraiton files

holiday configuration file 255Configuration files 255configuring custom groups 123, 175, 177, 179, 181, 267configuring custom resources 124, 274configuring data collector 192configuring exit statuses 124, 186, 188, 264configuring holidays 121, 136, 164, 169, 171, 255, 301configuring JVM performance 310, 311

data collector 311web application 310

configuring logging 304configuring node details 275configuring nodes 114, 121, 136, 147, 149, 152, 155, 159, 161configuring parser 109, 195, 198configuring plot point decimation 315configuring prime time hours 122, 258, 259


Index

configuring SSL 318configuring the license server 306, 308consumable resource 362CPU 362custom group 362custom group configuration file 175custom groups 119, 123custom resource 362custom resources 119, 124custom_group_YYYY.conf 175, 267customresources.conf 274

Ddaily parsing 20, 362dashboard 362dashboards 21data collector 366data collector log file 128data reset 323dc.log 128decimation 315default custom group 175, 267demonstration database 363deployment options 7

demonstration database 7, 11separate servers 7, 10single server 7, 9

dimension 363dimensions 211disaster recovery 325downsampling 315

Eexecution host 363exit status 119exit statuses 124exit_status.conf 264

FfactorySettings.xml 306, 307finished jobs 363

furnishing server 363

Ggroup 363

Hhardware requirements 2

CPUs 2disk space 2licensing 2RAM 2

head node 363High Performance Computing 363holiday 364holiday configuration file 164, 169, 171holiday_YYYY.conf 164holidays 119, 136holidays_YYYY.conf 255, 301hooks 335host 364http 301, 303, 317, 319https 301, 303, 317, 319

Iinitial parsing 19, 136, 364installation 13installation directory 105installing PBSA 25, 42, 55, 65, 79, 91

console mode 65, 79, 91separate servers 79, 91

install data collector 91install web application 79

single server 65swing mode 25, 42, 55

separate servers 42, 55install data collector 55install web application 42

single server 25


Index

Jjob

batch 364identifier 364owner 365state 364

job ID 364

Kkeystore file 318

Llicense server 364log file size 304log4j.properties 304logging 304logging backup files 304logging behavior 304

Mmeasure 364measures 211, 368migrating 21migration 359MOM 365Mother Superior 365

Nnode

definition 365node configuration 114, 121, 136node details 365node details configuration file 147, 149, 152, 155, 159, 161nodeclass 365nodeclass metrics 20nodedetails_YYYY.conf 147, 275non-consumable resource 365non-prime time 365, 366non-prime time hours 119, 122

OOTHERS 267owner 365

Pparser 366parser log file 128parser.log 128parsing 19, 20, 131, 362, 364

daily 20initial 19

parsing accounting logs 128PBS administrator 361PBS Analytics 365PBS Analytics™ 1PBS Professional 366PBS Professional 11.2 175PBS Professional considerations 20

application metrics 15nodeclass metrics 20project metrics 15

PBS Works 366PBS_EXEC 366PBS_HOME 366PBSA 365PBSA administrator 361PBSA Commands 109, 122, 123, 124, 136, 147, 149, 152, 155, 159, 161, 164, 169, 171, 175, 177, 179, 181, 186, 188, 192, 195, 198, 201, 202, 206, 207

change database password 206, 207configuring custom groups 123, 175,

177, 179, 181configuring data collector 192configuring exit statuses 124, 186, 188configuring holidays 136, 164, 169,

171configuring nodes 136, 147, 149, 152,


Index

155, 159, 161configuring parser 195, 198configuring prime time hours 122configuring the parser 109resetting PBSA dataset 201, 202

PBSA components 7data collector 7web application 7

PBSA configuration files 124custom resource 124

PBSA data collector 366PBSA engine 366PBSA hook 366PBSA parser 366PBSA service commands 141, 143, 144, 145

restarting data collector 144, 145restarting web application 141, 143starting data collector 144starting web application 141status of data collector 145status of web application 143stopping data collector 144stopping web application 141

PBSA service user 366PBSA web application 366pbsa-cofig-dc error messages 192, 198pbsa-config-dc 192pbsa-config-dc examples 192pbsa-config-exits 124, 186, 188pbsa-config-exits error messages 188pbsa-config-exits examples 186pbsa-config-groups 123, 175, 177, 179, 181pbsa-config-groups error messages 181pbsa-config-groups examples 179pbsa-config-groups usage 177pbsa-config-holidays 121, 122, 136, 164, 169, 171pbsa-config-holidays error messages 171pbsa-config-holidays examples 169pbsa-config-holidays usage 164pbsa-config-nodes 136, 147, 149, 152, 155,

159, 161pbsa-config-nodes error messages 161pbsa-config-nodes examples 152, 159pbsa-config-nodes usage 149, 155pbsa-config-parser 109, 195, 198pbsa-config-parser examples 198pbsa-config-parser usage 195pbsa-data-password 206, 207pbsa-data-password error messages 207pbsa-data-password uage 206pbsa-data-reset 201, 202pbsa-data-reset error messages 202pbsa-data-reset examples 202, 206pbsa-data-reset usage 201pbs-config-nodes 114, 121pbsnodes -av 114, 121, 147, 149, 152, 155, 159, 161pbsnodes.out 114, 121PBSWORKS_CONF_FILE 105pbsworks-dc 144, 145pbsworks-dc-start.sh 311pbsworks-pbsa 141, 143pbsworks-pbsa.conf 105Peer scheduling 366peer scheduling 333port 301, 303, 317, 319post-installation 105, 109, 114, 119, 121, 122, 123, 124, 128, 131, 136, 315

advanced configuration 109, 315configuration 114, 121, 122, 123, 124,

136parsing accounting logs 128, 136PBSA environment 105starting PBSA 109, 128, 136

pre-installation planning 13, 15, 19, 20configuration 19installation 13parsing 19, 20PBS Professional considerations 15, 20

prime time hours 119, 122Project 367project level utilization 175


Index

project metrics 15

Qqsub -P 175queue

definition 367

Rreparsing 323resetting PBSA dataset 201, 202Resource 367resource

consumable 362custom 362non-consumable 365

restarting data collector 144, 145restarting web application 141, 143

SScheduler 367

policies 367Server 367server-start.sh 310slideshows 21SSL 301, 303, 317, 319SSL certification 318start 127starting data collector 109, 144starting PBSA 109, 128starting web application 109, 141state

job 364status of data collector 145status of web 143Still-Running 264stopping data collector 144stopping web application 141supported platforms 4

browsers 4operating systems 4

system requirements 2

Ttroubleshooting 227

UUID 368uninstalling PBSA 99

swing mode 99user

definition 368ID 368

User charts 21

Vvalues 368vnode 368

Wweb appliaction 366


Index
