ibm loadleveler for linux -...

IBM LoadLeveler for Linux

Installation GuideVersion 5 Release 1

SC23-6789-00

��

NoteBefore using this information and the product it supports, read the information in “Notices” on page 35.

This edition applies to version 5, release 1, modification 0 of IBM LoadLeveler for Linux (product numbers5725-G01, 5641-LL1, and 5641-LL3) and to all subsequent releases and modifications until otherwise indicated innew editions.

© Copyright 1986, 1987, 1988, 1989, 1990, 1991 by the Condor Design Team.

© Copyright IBM Corporation 2010, 2012.US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contractwith IBM Corp.

Contents

Tables . . . . . . . . . . . . . . . v

About this information . . . . . . . . viiWho should read this information. . . . . . . viiConventions and terminology used in thisinformation . . . . . . . . . . . . . . viiPrerequisite and related information . . . . . . viiiHow to send your comments . . . . . . . . ix

Summary of changes. . . . . . . . . xi

Chapter 1. Introduction . . . . . . . . 1

Chapter 2. Pre-installation planning . . . 3Installation overview . . . . . . . . . . . 3

About the LoadLeveler RPMs. . . . . . . . 3Configuration planning . . . . . . . . . . 4Installing in a diskless environment. . . . . . 6General installation considerations . . . . . . 6

Migration considerations . . . . . . . . . . 7Migrating LoadLeveler . . . . . . . . . . 7

Upgrading considerations . . . . . . . . . . 9LoadLeveler APIs . . . . . . . . . . . . 10Hardware, software, and disk requirements. . . . 11

Hardware requirements . . . . . . . . . 11Software requirements . . . . . . . . . . 11Disk space requirements . . . . . . . . . 15

Configuration examples . . . . . . . . . . 16

Chapter 3. Installing LoadLeveler forLinux . . . . . . . . . . . . . . . 19Decide on directories for LoadLeveler . . . . . 19Log in as root . . . . . . . . . . . . . 19Create the loadl group name . . . . . . . . 19Create the loadl user ID . . . . . . . . . . 19Install the LoadLeveler for Linux RPMs . . . . . 20Run the LoadLeveler initialization script llinit(optional) . . . . . . . . . . . . . . . 22

Make the LoadLeveler for Linux man pagesavailable (optional) . . . . . . . . . . . . 23Set the default level for the llstatus and llrstatuscommands (optional) . . . . . . . . . . . 23Repeat the appropriate steps for each LoadLevelerfor Linux machine . . . . . . . . . . . . 24Configure your LoadLeveler for Linux system . . . 25

Chapter 4. Installing submit-onlyLoadLeveler for Linux . . . . . . . . 27Submit-only LoadLeveler for Linux planningconsiderations . . . . . . . . . . . . . 27

Submit-only LoadLeveler for Linux disk spacerequirements . . . . . . . . . . . . . 27

Set up the submit-only LoadLeveler user ID andgroup . . . . . . . . . . . . . . . . 27Install the submit-only LoadLeveler for Linux RPMs 28Set up your submit-only LoadLeveler configuration 28Update the submit-only LoadLeveler clusterconfiguration . . . . . . . . . . . . . . 29

Updating File-based configuration . . . . . . 29Updating Database-based configuration . . . . 30

Install submit-only LoadLeveler for Linux on othermachines in the pool . . . . . . . . . . . 30Running reservation commands without startingLoadLeveler . . . . . . . . . . . . . . 31

Accessibility features for LoadLeveler 33Accessibility features . . . . . . . . . . . 33Keyboard navigation . . . . . . . . . . . 33IBM and accessibility . . . . . . . . . . . 33

Notices . . . . . . . . . . . . . . 35Trademarks . . . . . . . . . . . . . . 37

Glossary . . . . . . . . . . . . . . 39

Index . . . . . . . . . . . . . . . 43

© Copyright IBM Corp. 2010, 2012 iii

iv LoadLeveler for Linux: Installation Guide

Tables

1. Conventions . . . . . . . . . . . . vii2. LoadLeveler for Linux RPMs and their

associated directories . . . . . . . . . . 43. Linux platforms and associated identifiers 44. LoadLeveler information and default values for

file-based configuration . . . . . . . . . 55. LoadLeveler information and default values for

database-based configuration . . . . . . . 66. LoadLeveler for Linux on x86 Architecture and

LoadLeveler for Linux on POWER softwarerequirements . . . . . . . . . . . . 12

7. Information about LoadLeveler for Linuxfacilities . . . . . . . . . . . . . . 15

8. LoadLeveler configuration example . . . . 169. Sample LoadL_admin and LoadL_config file

entries . . . . . . . . . . . . . . 17

10. LoadLeveler database-based configurationexample. . . . . . . . . . . . . . 17

11. Sample LoadL_admin and LoadL_config fileentries for loading database . . . . . . . 18

12. LoadLeveler for Linux directory locationchecklist . . . . . . . . . . . . . 19

13. Procedure for viewing and accepting thelicense . . . . . . . . . . . . . . 21

14. Configuration considerations when thescheduler component is not installed on allmachines . . . . . . . . . . . . . 25

15. Submit-only LoadLeveler for Linux disk spacerequirements . . . . . . . . . . . . 27

16. Submit-only LoadLeveler for Linux directoryinformation . . . . . . . . . . . . 28

© Copyright IBM Corp. 2010, 2012 v

vi LoadLeveler for Linux: Installation Guide

About this informationAttention:

For LoadLeveler® Version 5 Release 1, changes apply to Linux.

Disclaimer:

The functions or features found herein may not be available on all operating systems orplatforms and do not indicate the availability of these functions or features within theIBM® product or future versions of the IBM product. The development, release, and timingof any future features or functionality is at IBM's sole discretion. IBM's plans, directions,and intent are subject to change or withdrawal without notice at IBM's sole discretion. Theinformation mentioned is not a commitment, promise, or legal obligation to deliver anymaterial, code or functionality. The information may not be incorporated into any contractand it should not be relied on in making a purchasing decision.

This information will help you install, configure, and migrate IBM LoadLeveler. Itincludes concepts and instructions pertaining to:v Installing and configuring LoadLeveler for Linuxv Installing and configuring submit-only LoadLeveler for Linux

Who should read this informationThis information is intended for system administrators responsible for installing,configuring, and maintaining the LoadLeveler system. It assumes that theadministrators have a working knowledge of the Linux operating systems.

Conventions and terminology used in this informationThroughout the IBM LoadLeveler product information:v LoadLeveler for Linux on x86 Architecture includes:

– IBM System x® Intelligent Cluster™

– IBM System servers with Advanced Micro Devices (AMD) Opteron or Intel®

Extended Memory 64 Technology (EM64T) processorsv References to Schedd are also referred to as job manager.

Table 1. Conventions

Convention Usage

bold Bold words or characters represent system elements that you mustuse literally, such as commands, flags, path names, directories, filenames, values, and selected menu options.

bold underlined Bold underlined keywords are defaults. These take effect if you donot specify a different keyword.

constant width Examples and information that the system displays appear inconstant-width typeface.

italic Italic words or characters represent variable values that you mustsupply.

Italics are also used for information unit titles, for the first use of aglossary term, and for general emphasis in text.

© Copyright IBM Corp. 2010, 2012 vii

Table 1. Conventions (continued)

Convention Usage

<key> Angle brackets (less-than and greater-than) enclose the name of akey on the keyboard. For example, <Enter> refers to the key onyour terminal or workstation that is labeled with the word Enter.

\ In command examples, a backslash indicates that the command orcoding example continues on the next line. For example:

mkcondition -r IBM.FileSystem -e "PercentTotUsed > 90" \-E "PercentTotUsed < 85" -m d "FileSystem space used"

{item} Braces enclose a list from which you must choose an item in formatand syntax descriptions.

[item] Brackets enclose optional items in format and syntax descriptions.

<Ctrl-x> The notation <Ctrl-x> indicates a control character sequence. Forexample, <Ctrl-c> means that you hold down the control key whilepressing <c>.

item... Ellipses indicate that you can repeat the preceding item one or moretimes.

| v In syntax statements, vertical lines separate a list of choices. Inother words, a vertical line means Or.

v In the left margin of the document, vertical lines indicatetechnical changes to the information.

Prerequisite and related informationThe LoadLeveler publications are:v AIX Installation Guide, SC23-6791v Linux Installation Guide, SC23-6789v Using and Administering, SC23-6792v Diagnosis and Messages Guide, SC23-6793v Command and API Reference, SC23-6794v Resource Manager, SC23-6790

To access all LoadLeveler documentation, refer to the IBM Cluster InformationCenter (http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp), whichcontains the most recent LoadLeveler documentation in PDF and HTML formats.

A LoadLeveler Documentation Updates file also is maintained on this Web site.The LoadLeveler Documentation Updates file contains updates to the LoadLevelerdocumentation. These updates include documentation corrections and clarificationsthat were discovered after the LoadLeveler information units were published.

Both the current LoadLeveler books and earlier versions of the library are alsoavailable in PDF format from the IBM Publications Center (http://www.ibm.com/e-business/linkweb/publications/servlet/pbi.wss).

To easily locate a book in the IBM Publications Center, supply the book'spublication number. The publication number for each of the LoadLeveler books islisted after the book title in the preceding list.

viii LoadLeveler for Linux: Installation Guide

http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp


http://www.ibm.com/e-business/linkweb/publications/servlet/pbi.wss

How to send your comments

Your feedback is important in helping us to produce accurate, high-qualityinformation. If you have any comments about this book or any other LoadLevelerdocumentation, send your comments by e-mail to:

[email protected]

Include the book title and order number, and, if applicable, the specific location ofthe information you have comments on (for example, a page number or a tablenumber).

For technical information and to exchange ideas related to high performancecomputing, go to:v HPC Central (http://www.ibm.com/developerworks/wikis/display/

hpccentral/HPC+Central)v HPC Central Technical Forum (http://www.ibm.com/developerworks/forums/

forum.jspa?forumID=1056)

About this information ix

http://www.ibm.com/developerworks/wikis/display/hpccentral/HPC+Central

http://www.ibm.com/developerworks/forums/forum.jspa?forumID=1056

x LoadLeveler for Linux: Installation Guide

Summary of changes

Version 5 Release 1

The following sections summarize changes to the IBM LoadLeveler product andlibrary.

Within each information unit in the library, a vertical line to the left of text andillustrations indicates technical changes or additions made to the previous editionof the information.

Attention:

For LoadLeveler Version 5 Release 1, changes apply to Linux.

Disclaimer:

The functions or features found herein may not be available on all operating systems orplatforms and do not indicate the availability of these functions or features within the IBMproduct or future versions of the IBM product. The development, release, and timing ofany future features or functionality is at IBM's sole discretion. IBM's plans, directions, andintent are subject to change or withdrawal without notice at IBM's sole discretion. Theinformation mentioned is not a commitment, promise, or legal obligation to deliver anymaterial, code or functionality. The information may not be incorporated into any contractand it should not be relied on in making a purchasing decision.

Changes to LoadLeveler for Linux Version 5 Release 1 product and library include:v New information:

– A new affinity interface is introduced into LoadLeveler. The new interface isused by IBM Parallel Environment (PE) Runtime Edition on the x86 platform.

– Island scheduling is introduced in LoadLeveler. New configurationparameters permit administrators to define islands within a LoadLevelercluster. You can now request that your jobs run within one island or acrossmultiple islands.

– Support for running Intel Message Passing Interface (MPI) and Open MPIjobs as batch LoadLeveler jobs has been added.

– The llrun command has been added to allow you to run an interactivecommand that uses resources allocated by LoadLeveler.

– LoadLeveler provides support for running embarrassingly parallel jobswithout using an MPI runtime process manager. For more information, seethe following:- LoadLeveler: Command and API Reference

- LoadLeveler: Using and Administering

– Support for checkpoint/restart on Red Hat Enterprise Linux (RHEL) 6.2,which includes the following:- The system administrator or user can now set the checkpoint interval on a

per job basis.- Task migration, which provides a method to move portions of a running

parallel job from the current allocated set of nodes to a new set of nodes.To support task migration, the following have been added:v The llmigrate command

© Copyright IBM Corp. 2010, 2012 xi

v Two new migration subroutines, ll_migrate_job_step andllr_start_job_migration

For more information, see the following:- LoadLeveler: Command and API Reference

- LoadLeveler: Resource Manager


– Support for running LoadLeveler on Blue Gene/Q has been added onLoadLeveler for Linux on POWER® (RHEL 6.2). For more information, see thefollowing:- LoadLeveler: Command and API Reference


- LoadLeveler: Resource Manager

v Changed information:

– References to virtual IP (VIP) addresses or the VIP server are for AIX® only.– The following table summarizes the Blue Gene/Q terminology changes:

BG/L and BG/P (previous terms) BG/Q terms

Partitions Blocks

Base partitions Midplanes

Wires Cables

Node cards Node boards

mpirun, submit, mpiexec runjob

v Deleted information:

– The ALLOC_EXCLUSIVE_CPU_PER_JOB configuration file keyword hasbeen deprecated and has been removed.

– The following Blue Gene® job command file keywords have been deprecatedand removed:- bg_connection (replaced with bg_connectivity)- bg_partition (replaced with bg_block)

xii LoadLeveler for Linux: Installation Guide

Chapter 1. Introduction

Installation descriptions encompass a range of product and processor options.

This topic describes how to install LoadLeveler for Linux Version 5 Release 1.0running on:v Red Hat Enterprise Linux (RHEL 6.2) on LoadLeveler for Linux on x86

Architecturev SUSE Linux Enterprise Server (SLES) 11 on Linux LoadLeveler for Linux on x86

Architecture

References to LoadLeveler Version 5 Release 1 apply to the latest version ofLoadLeveler, which is 5.1.0.x, where x represents the latest service update. At thetime this information unit was published, the current service update was 5.1.0.4.The version of the base RPMs is 5.1.0.3. To find out which release of LoadLeveleryou currently are running, issue the llctl version command.

You can obtain Linux service updates at Fix Central (http://www-933.ibm.com/support/fixcentral/?productGroup0=ibm/fcpower&productGroup1=ibm/ClusterSoftware&productGroup2=ibm/power/IBM).

© Copyright IBM Corporation © IBM 2010, 2012 1

http://www-933.ibm.com/support/fixcentral/?productGroup0=ibm/fcpower&productGroup1=ibm/ClusterSoftware&productGroup2=ibm/power/IBM

2 LoadLeveler for Linux: Installation Guide

Chapter 2. Pre-installation planning

Before installing LoadLeveler, there are planning activities that you must firstconsider.

This topic explains the things you should be aware of before you installLoadLeveler.

Installation overviewBefore installing LoadLeveler, take these steps to make the necessaryconsiderations and decisions.

Complete the following steps to install LoadLeveler:1. Ensure prerequisites are installed. (See “Software requirements” on page 11.)2. Make configuration decisions, including:

v Decide which machines will act as the following types of managers:– central manager– resource manager– backup managers for the central manager and the resource manager

(optional)– region manager (optional)

v Decide on the locations of the home and local directories for the LoadLeveleruser ID

v Decide which directories will be shared directories, and which will not beshared.

v Decide whether the file or database option will be used for configuration. Ifdatabase-based configuration will be used, determine which nodes willconnect to the database and which will get the configuration from othernodes.

v Decide whether the llstatus and llrstatus commands should show machine,machine group, or cluster level output. The default is cluster level.

3. Set up the LoadLeveler user ID and group ID for all of the machines in theLoadLeveler cluster.

4. Install the LoadLeveler for Linux RPM packages and any mandatory PTFs.5. Optional step: Run the initialization script.6. Update the configuration.7. Optional step: If using the database option, populate the configuration data

using the llconfig command.8. Repeat the installation process for each workstation in the LoadLeveler cluster.

About the LoadLeveler RPMsRPMs comprise the LoadLeveler for Linux product.

Table 2 on page 4 shows the RPMs that comprise the LoadLeveler for Linuxproduct. It also shows their associated Linux directories after installation.


Table 2. LoadLeveler for Linux RPMs and their associated directories

RPM Description Associated directory

LoadL-full-license-Linux_identifier.rpm LoadLeveler license acceptance /opt/ibmll/LoadL/lap/opt/ibmll/LoadL/sbin

LoadL-resmgr-full-Linux_identifier.rpm LoadLeveler Resource Manager /opt/ibmll/LoadL/resmgr

LoadL-resmgr-kbdd-Linux_identifier.rpm Keyboard Daemon forLoadLeveler

/opt/ibmll/LoadL/resmgr

LoadL-scheduler-full-Linux_identifier.rpm LoadLeveler Scheduler /opt/ibmll/LoadL/scheduler

See table note.

LoadL-scheduler-bluegene-Linux_identifier.rpm

LoadLeveler Scheduler supportfor Blue Gene®/Q

/opt/ibmll/LoadL/scheduler

LoadL-utils-Linux_identifier.rpm Utilities for LoadLeveler /var/www/ll/

This package contains theConfiguration Editor files

LoadL-so-Linux_identifier.rpm Submit-only LoadLeveler /opt/ibmll/LoadL/scheduler/so/

See table note.

Table Note: Symbolic links are created from various directories in /opt/ibmll/LoadL/full or /opt/ibmll/LoadL/so to/opt/ibmll/LoadL/scheduler/full or /opt/ibmll/LoadL/scheduler/so.

Table 3 shows the supported Linux identifiers:

Table 3. Linux platforms and associated identifiers

Platform Identifier

RHEL 6.2 on AMD Opteron or Intel EMT 64-bitplatforms

RH6-X86_64-5.1.0.v-0.x86_64.rpm

RHEL 6.2 on POWER systems RH6-PPC64-5.1.0.v-0.ppc64.rpm

SLES 11 on AMD Opteron or Intel EMT 64-bitplatforms

SLES11-X86_64-5.1.0.v-0.x86_64.rpm

Table Note: v represents the latest service update version.

Configuration planningAn integral part of the installation of LoadLeveler is to configure your nodes. Yourconfiguration can be file based or in a database.

LoadLeveler assumes a common name space. This means that you need the sameuser ID and group ID on all machines in a LoadLeveler cluster. The term clusterrefers to all machines mentioned in the LoadLeveler configuration file. SeeLoadLeveler: Using and Administering for planning considerations for definingmachines.

You can define your own LoadLeveler user ID and group ID. Note, however, thatif you use an ID other than loadl, you also have to do the following:v Substitute this new ID into all appropriate installation instructions.v Create a master configuration file (usually called /etc/LoadL.cfg) specifying the

user ID and group ID, and place it in the local directory /etc of each of thenodes in your LoadLeveler cluster.


Note: For information about defining your own LoadLeveler user ID and groupID, see “Create the loadl user ID” on page 19.

You can also use the master configuration file to specify the source of theconfiguration. How you specify this depends on whether the configuration isfile-based (see “File-based configuration”) or database-based (see “Database-basedconfiguration”):v For file-based configuration, a master configuration file is optional.

A global configuration file, LoadL_config, is used and resides in theLoadLeveler user ID's home directory. To specify a different location for this file,use the LoadLConfig keyword in the master configuration file.

v For database-based configuration, a master configuration file is required onevery node to indicate the source of the configuration.– On nodes that have access to the database, specify the data source name in

the LoadLDB keyword. These nodes will get the configuration from thedatabase.

– On nodes without database access, specify the LoadLConfigHosts keywordand assign it nodes that have the LoadLDB keyword in their masterconfiguration file. The nodes specifying LoadLConfigHosts will get theconfiguration from one of these nodes that can access the database.

See LoadLeveler: Using and Administering and LoadLeveler: Resource Manager for fullconfiguration details.

File-based configuration

The keywords LOCAL_CONFIG and ADMIN_FILE in the global configurationfile define the full path names of the local configuration and the administrationfiles. The global configuration file specifies system attributes common to all nodes.The administration file specifies attributes specific to individual nodes. Theinformation given in the local configuration file is specific to an individual node. Itoverrides the corresponding information given in the global configuration file.

Table 4. LoadLeveler information and default values for file-based configuration

File-based configuration information Default value

LoadLeveler user ID and group ID loadl is the LoadLeveler user ID and theLoadLeveler group ID

The configuration directory The home directory of loadl is theconfiguration directory

The global configuration file LoadL_config is the name of theconfiguration file

Database-based configuration

For database-based configuration, a master configuration file is required on eachnode. One of the keywords LoadLDB or LoadLConfigHosts is needed in themaster configuration file to define whether the node will access the configurationdatabase or rely on other nodes to access the configuration database.

To initialize the configuration in the database, llconfig -i is run by root on aLoadLeveler node that can access the database. The initial settings are based on aglobal configuration file and the LoadL_admin file it specifies. The location of theglobal configuration file can be specified, or the default can be used.

Chapter 2. Pre-installation planning 5

Table 5. LoadLeveler information and default values for database-based configuration

Database-based configuration information Default value

LoadLeveler user ID and group ID loadl is the LoadLeveler user ID and theLoadLeveler group ID

The configuration database The MySQL database or DB2® configured forxCAT in /etc/xcat/cfgloc

The master configuration file /etc/LoadL.cfg is the name of theconfiguration file on each node havingLoadLDB set for nodes which will accessthe database and LoadLConfigHosts set fornodes that will contact other nodes to getthe configuration.

Installing in a diskless environment

For information about installing LoadLeveler in a diskless environment, see:

Setting up the IBM HPC Stack in an xCAT Cluster (Product-Specific Information)(http://sourceforge.net/apps/mediawiki/xcat/index.php?title=Setting_up_the_IBM_HPC_Stack_in_an_xCAT_Cluster)

General installation considerationsWhen planning your LoadLeveler installation, there are specific activities that youmust first consider.v Generally, both the resmgr and scheduler components are installed on all

machines in a cluster, so the steps in this information unit are meant forinstalling them both. It is possible to install the resource manager componentwithout installing the scheduler component, but it requires specializedconfiguration. See "Configuration considerations when the scheduler componentis not installed on all machines" in the topic, “Configure your LoadLeveler forLinux system” on page 25 for more information.

v Many of the steps for installing LoadLeveler must be performed as root.Therefore, you need superuser authority on each machine in the cluster.

v Linux commands are case sensitive. Therefore, you must enter commands,directory names, and so on exactly as they appear.

v Many of the installation steps contain examples that assume you are usingstandard LoadLeveler directories and names. If you do not use the standardnames, you must modify the examples accordingly. This is especially true for theLoadLeveler local and home directories.

v For file-based configuration, the directory containing the configuration files mustbe readable by all users who will submit jobs.

v For database configuration, the database space will be shared with xCAT.v To use the llctl start command to start remote machines in a cluster, all

LoadLeveler administrator user IDs must have rsh privileges on all machines inthe cluster. Check the appropriate Linux documentation for more information onthe files used by rsh, such as $HOME/.rhosts and /etc/hosts.equiv.If you use the LL_RSH_COMMAND keyword to specify using anotherexecutable instead of rsh, ensure all appropriate files and necessary permissionshave been set up.

v The LoadL_master binary is installed as a setuid program with the owner set toroot. If the release directory will be shared among machines in the cluster using


http://sourceforge.net/apps/mediawiki/xcat/index.php?title=Setting_up_the_IBM_HPC_Stack_in_an_xCAT_Cluster

NFS, you must allow all files to be world-readable (by default), or you mustexport the file system with root permission to all hosts that will be runningLoadLeveler. When mounting the file system, do not use the -nosuid flag.

v LoadLeveler log files should be stored locally at the nodes for best performance.v Any machine designated as a region manager needs to have access to all

networks of the compute machines that it is monitoring, and it also needs to beable to send transactions to the resource manager.

Migration considerationsThis topic discusses considerations for migrating to LoadLeveler for Linux fromthe previous release.

Note: If you are running LoadLeveler in a mixed cluster, refer to the appropriatelevel of documentation for each version of LoadLeveler in the cluster.

Use either of the following two methods to migrate LoadLeveler:1. Take down all of the machines and migrate the entire cluster at once.2. Migrate your cluster gradually one machine or a group of machines at a time.

This method requires that the first machine in the cluster to be migrated is thecentral manager.

Whichever method you choose, you should make backup copies of the followingfiles:v LoadL_admin, the LoadLeveler administration filev LoadL_config, the LoadLeveler global configuration filev LoadL_config.local, the LoadLeveler local configuration file

Note: Although LoadLeveler supports job queues from the previous release,parallel jobs that specify poe as the executable with a statement such as # @executable = /bin/poe in a job command file cannot be migrated. When those jobsare submitted, the executable poe is copied to LoadLeveler's execute directory andis used when the job runs. Since the previous version of poe copied before themigration is not compatible with the current release of LoadLeveler, the job willfail when it runs.

You can circumvent this by not using the executable keyword in the job commandfile, but rather using the job command file as a shell script invoking the POEbinary.

Migrating LoadLevelerFollow either of these step-by-step instructions depicting common migrationscenarios.

“Scenario 1: Migrating an entire LoadLeveler cluster” requires that all of themachines in the cluster be migrated together. “Scenario 2: Machine-by-machineLoadLeveler migration” on page 8 corresponds to a machine-by-machine migrationstrategy.

Scenario 1: Migrating an entire LoadLeveler clusterThis scenario requires that all of the machines in the cluster be migrated together.1. Log in as a LoadLeveler administrator. Issue the command llctl -g drain. This

will drain all of the Schedd machines so that no new jobs can be submitted to


LoadLeveler. This will also drain all of the startd machines so that no jobsalready submitted to LoadLeveler can be started. Jobs that are currentlyrunning will continue to run.

2. After all of the running jobs have completed and only idle jobs remain in thejob queues, issue the command llctl -g stop to stop LoadLeveler on all of themachines in the cluster.

3. Run the llsummary command on the machines that run the Schedd daemons ifaccounting reports are required. Accounting reports are made by first runningthe llacctmrg command to create a global history file before running thellsummary command.

4. Perform a backup of all of the files in the LoadLeveler spool directory on allmachines that run a Schedd daemon. This step is not essential, but is stronglysuggested in case a failure occurs during the installation and the previousrelease needs to be reinstalled along with the backup spool directory files.

5. Install the new version of LoadLeveler using the instructions described inChapter 3, “Installing LoadLeveler for Linux,” on page 19. You will need rootpermission to perform this operation.

6. If you are using database configuration, run perl /usr/bin/lldbupdate on themachine running the database. This will make any necessary updates to theLoadLeveler database schema.

7. Issue the command llctl -g start to bring up the migrated machines, or if usingdatabase-based configuration, start LoadLeveler on the configuration hosts first,then on the rest of the cluster.

Scenario 2: Machine-by-machine LoadLeveler migrationThis scenario corresponds to a machine-by-machine migration strategy.

When xCAT is used for system management, you can take advantage of the xCATrolling update support to automate migration. See the xCAT Rolling UpdateSupport documentation (http://sourceforge.net/apps/mediawiki/xcat/index.php?title=Rolling_Update_Support) for more information.

Being able to migrate one machine at a time implies that the new release ofLoadLeveler can coexist with the preceding LoadLeveler release. In other words, itis possible to set up a LoadLeveler cluster consisting of some machines runningLoadLeveler 5.1 level of software while the rest of the cluster remains at theLoadLeveler 4.1 software level.

When migrating this way, some machines must be migrated before others. Theorder of migrating is determined by which daemons are running on the machines.A simple rule of thumb is to migrate all the machines running manager daemonsfirst. This includes the central manager, resource manager, and region managers.Or you can migrate them in this order:1. Central and resource manager machines2. Region manager machines3. Other machines

Note: If you run in this environment, it is possible that some LoadLevelercommands will display message 2539-433 referring to ** unknownLL_Specification ** multiple times. This message can be ignored. The messagewill not be displayed after all machines in the cluster are upgraded to the newrelease.

To migrate machine-by-machine:


http://sourceforge.net/apps/mediawiki/xcat/index.php?title=Rolling_Update_Support


1. Select the machines that you want to migrate. Remember, the central managerand resource manager machine or machines must be migrated before any othermachines can be restarted with the new LoadLeveler release. Log in as aLoadLeveler administrator. On machines running Schedd or Startd daemons,issue the command llctl -h machine_name drain. As a result of this action, nonew jobs can be submitted to this machine and no new jobs will be started onthis machine. Jobs that are currently running will continue to run. Repeat thisstep for all of the machines to be migrated.

2. After all running jobs on the machines to be migrated have completed and onlyidle jobs remain in the queues of the Schedd daemons on the machines to bemigrated, issue the command llctl -h machine_name stop to stop LoadLeveler onthe specified machine. Repeat this action for all of the machines to be migrated.

3. Perform a backup of all of the files in the LoadLeveler spool directory on allmachines that run a Schedd daemon. This step is not essential, but is stronglysuggested in case a failure occurs during the installation and the previousrelease needs to be reinstalled along with the backup spool directory files.

4. Install the new version of LoadLeveler using the instructions described inChapter 3, “Installing LoadLeveler for Linux,” on page 19. You will need rootpermission to perform this operation.

5. Issue the command llctl -h machine_name start to bring up LoadLeveler 5.1 onthe specified machine. Repeat this action for all of the machines to be migrated.At this point, the migrated machines should be running LoadLeveler 5.1 andany jobs carried over will run in the upgraded cluster.

New features in LoadLeveler 5.1 cannot be used in a mixed cluster of LoadLeveler5.1 and 4.1 machines. If you are running LoadLeveler in a mixed cluster, refer tothe appropriate level of information for each version of LoadLeveler in the cluster.

If you are using database configuration, you will need to run perl/usr/bin/lldbupdate on the machine running the database.

Upgrading considerationsUpgrading LoadLeveler to a later service pack requires stopping LoadLeveler onthe nodes in an orderly manner, applying the upgrades, and restartingLoadLeveler.

As a simple rule of thumb, upgrade the machines running the manager daemonsand their alternates first, before upgrading the executing machines. In this context,the manager daemons include the central manager daemon (LoadL_negotiator), theresource manager daemon (LoadL_resource_mgr), and any region manager(LoadL_region_mgr).

If you are using database configuration, you will need to run perl/usr/bin/lldbupdate on the machine running the database. After the centralmanager and resource manager machines are updated, run lldbupdate before themachine or machines are restarted.

To automate upgrading LoadLeveler on a large cluster, use xCAT rolling updatessupport. See the Rolling Update Support documentation (http://sourceforge.net/apps/mediawiki/xcat/index.php?title=Rolling_Update_Support) for moreinformation.

Before you begin, be sure:



v That the central manager, resource manager, and region manager daemons haveone or more corresponding alternate daemons

v There are two or more job managers in the cluster

Orderly shutdown of a node

The role of the node in the cluster determines some steps and considerations thatmight be needed before stopping LoadLeveler on the node.1. Consider the LoadLeveler daemons configured to run on this node:

For nodes running central manager (LoadL_negotiator), resource manager(LoadL_resource_mgr), or region manager (LoadL_region_mgr), ensure thereare nodes set up to run alternate manager daemons while this daemon isdown.For nodes running a job manager (LoadL_schedd):v Ensure that there are enough schedd daemons available to maintain a

running cluster while this machine is down.v Drain the machine, by entering:

llctl -h machinename drain

v Optionally, move the job queue to another schedd machine by entering:llmovespool -d directory -h new_running_schedd

Any idle jobs still in the spool will not run while schedd is down. If the jobsneed to run before this node is restarted, run the llmovespool command.

For nodes running a startd daemon (LoadL_startd):v Drain the machine by entering:

llctl -h machinename drain

v Wait for running jobs to complete, or alternately, if the jobs can be terminatedand restarted, flush the startd by entering:llctl -h machinename flush

2. If database configuration is being used and this node is a configuration host,ensure that stopping this node does not result in other nodes losing access tothe configuration.If the node you are stopping is specified by the LoadLConfigHosts list in/etc/LoadL.cfg on other machines in the cluster, those machines will be unableto get the configuration while this node is stopped, unless backup configurationhosts are specified in the list.

3. Shutdown LoadLeveler on your node, by entering:llctl -h machinename stop

LoadLeveler APIsLoadLeveler provides application programming interfaces (APIs) that allowapplication programs written by customers to use specific data or functions thatare part of LoadLeveler.

These APIs are enabled in such a way that user programs linked dynamically tothe library do not have to be compiled and linked again when an installationmigrates from one LoadLeveler release to the next.

Note that a user application compiled with an earlier LoadLeveler version islimited to the information specific to that release. Thus, if you have an applicationusing a LoadLeveler API compiled in a previous LoadLeveler release, you do not


absolutely have to recompile to use it with latest version of LoadLeveler. However,this application cannot access the new data and functions specific to the latestLoadLeveler release. In order to do this, you may have to make enhancements tothe source codes of your application and recompile this application on aLoadLeveler machine using the new version of the llapi.h header file.

Hardware, software, and disk requirementsPlanning for installation requires two main tasks: ensuring that you have thecorrect software and hardware installed, and allocating the correct amount of diskspace for directories.

Note: You are responsible for licensing and installing these products andintegrating them with LoadLeveler.

Hardware requirementsInstallation planning requires that you ensure that compatible hardware will beinstalled.

The following topics list the LoadLeveler hardware requirements.

LoadLeveler for Linux on x86 Architecture hardwarerequirements

LoadLeveler for Linux on x86 Architecture supports the following hardwareplatforms:v IBM System servers with AMD Opteron or Intel EM64T processorsv IBM System x servers, including iDataPlex®

v IBM System x Intelligent Cluster

LoadLeveler for Linux on x86 Architecture requires:v A minimum of 1 GB of memoryv 1 GB of swap space

LoadLeveler for Linux on POWER hardware requirements

LoadLeveler for Linux on POWER supports the following hardware platforms:v IBM Power Systems™ (POWER7®)v IBM OpenPower®

LoadLeveler for Linux on POWER requires:v A minimum of 1 GB of memoryv 1 GB of swap space

Software requirementsInstallation planning requires that you ensure that compatible software will beinstalled.

The following topics list the LoadLeveler software requirements.


LoadLeveler for Linux software requirementsLoadLeveler for Linux has certain software platform requirements.

Table 6 lists the LoadLeveler for Linux software requirements.

Table 6. LoadLeveler for Linux on x86 Architecture and LoadLeveler for Linux on POWER software requirements

To use: LoadLeveler for Linux on x86Architecture

LoadLeveler for Linux on POWER

Linux operating systems(requires the libelf package)

RHEL 6.2

SLES 11

RHEL 6.2

Mandatory LoadLevelerService

RPMs for 5.1.0.v

You can obtain Linux service updates at Fix Central (http://www-933.ibm.com/support/fixcentral/?productGroup0=ibm/fcpower&productGroup1=ibm/ClusterSoftware&productGroup2=ibm/power/IBM).

If you want:

Process tracking For RHEL 6.2 and SLES 11, you do not need to install any processing trackingkernel modules.

afs

(See note 1 on page 13.)

RHEL 6.2: OpenAFS 1.4.1 or later Not applicable

IBM System Blue GeneSolution

N/A RHEL 6.2

Parallel jobs Using:

POE IBM PE Runtime Edition 1.2.0

ppe.rte 1.2.0

MPICH21.3.2p1 or later

MVAPICHNot supported

Open MPI1.5.4 or later

Intel MPI4.0.2 or later

Checkpoint/Restart IBM PE Runtime Edition 1.2.0

ppe.rte 1.2.0

IBM Power® 775 server

OpenSSL in a multiclusterenvironment

OpenSSL 0.9.7 or later

RDMA with InfiniBandadapters

RHEL 6.2: ppe.rte 1.2.0

OpenMP thread binding Not applicable XL C or C++ 11.1.0.4 or later

Not applicable XL FORTRAN 13.1.04 or later

Database configuration option


xCAT 2.6.6 or later, including:

xCAT (xcat-core) tarballxCAT dependencies (xcat-deps) tarball

from:

SourceForge (http://sourceforge.net/projects/xcat/files)



http://sourceforge.net/projects/xcat/files/

Table 6. LoadLeveler for Linux on x86 Architecture and LoadLeveler for Linux on POWER softwarerequirements (continued)

To use: LoadLeveler for Linux on x86Architecture

LoadLeveler for Linux on POWER

Database configuration option(continued)

MySQL from the OS:

For a list of RPMs, see Setting UpMySQL as the xCAT DB

(http://sourceforge.net/apps/mediawiki/xcat/index.php?title=Setting_Up_MySQL_as_the_xCAT_DB)

Your licensed copy of DB2

See Setting Up DB2 as the xCAT DB

(http://sourceforge.net/apps/mediawiki/xcat/index.php?title=Setting_Up_DB2_as_the_xCAT_DB

Configuration editor fordatabase configuration


IBM HTTP Server V6.1 or Apache 2.0 HTTP Server, from:

IBM HTTP Server (http://www-01.ibm.com/software/webservers/httpservers/)

Apache (http://httpd.apache.org/)

xCAT 2.6.6 Web dependencies, detailed in the document:

Installing the xCAT Web Interface (http://sourceforge.net/apps/mediawiki/xcat/index.php?title=Installing_the_xCAT_Web_Interface).

CGI::Session 3.95 or later Perl Module, from:

CPAN (http://search.cpan.org/dist/CGI-Session)

lltrace command python 2.6 and dependencies from the Linux distribution

pyODBC from the LoadLeveler product CD

Adapter and node-statusheartbeat

ppe.rte 1.2.0

Keyboard daemon(LoadL_kbdd)

From the libXScrnSaver package:

libXss.so.1

Notes on the LoadLeveler for Linux software requirements:

1. OpenAFS 1.4.1 or later is supported on LoadLeveler for Linux nodes. It can beused for token passing and token refresh between Linux and AIX. OpenAFS isnot supported on LoadLeveler for AIX nodes. For more information aboutsetting up AFS® on Red Hat Linux, see the LoadL.*.README for AFS for theplatform, for example the README LoadL.resmgr.README.RH_X86_64_AFSfor RHEL 6.2.

2. If you plan to use the process tracking or preemption functions on RHEL 6.2 orSLES 11, or the checkpoint/restart function on RHEL 6.2, you must define afreezer control group on each machine in the cluster. You can do this by addingthe following line to /etc/fstab:cgroup /cgroup cgroup ns,freezer 0 0

and then run the following command:mount -a

If you plan to use both checkpoint/restart and scheduling affinity, you mustensure that /cgroup is always mounted before /dev/cpuset.


http://sourceforge.net/apps/mediawiki/xcat/index.php?title=Setting_Up_MySQL_as_the_xCAT_DB

http://sourceforge.net/apps/mediawiki/xcat/index.php?title=Setting_Up_MySQL_as_the_xCAT_DB

http://sourceforge.net/apps/mediawiki/xcat/index.php?title=Setting_Up_DB2_as_the_xCAT_DB

http://www-01.ibm.com/software/webservers/httpservers/

http://httpd.apache.org/

http://sourceforge.net/apps/mediawiki/xcat/index.php?title=Installing_the_xCAT_Web_Interface

http://search.cpan.org/dist/CGI-Session/

3. If you plan to use scheduling affinity, you must create a /dev/cpuset directoryusing the mkdir /dev/cpuset command. Then mount a cpuset file system to/dev/cpuset using the following command:mount -t cpuset none /dev/cpuset

4. If you plan to configure LoadLeveler to support the database configurationoption, first do the following:a. Ensure that xCAT is installed on the management node and is using

MySQL or DB2 as its database. Consult the following xCAT documentationfor more information:v Setting Up an xCAT Management Node

v Setting Up MySQL as the xCAT DB

v Setting Up DB2 as the xCAT DB

These documents are available from:XCAT Documentation (http://sourceforge.net/apps/mediawiki/xcat/index.php?title=XCAT_Documentation).When setting up MySQL, you also need to complete the following steps:

1) Ensure that all LoadLeveler nodes that will access the database areincluded when granting permissions in MySQL to access the database.

2) Set the MySQL log-bin-trust-function-creators variable to ON.You can set this variable using the --log-bin-trust-function-creators=1option when starting the server; or you can set it using the MySQLinteractive command:mysql> SET GLOBAL log_bin_trust_function_creators=1;

This allows LoadLeveler to create triggers and stored procedures forMySQL. This option is supported in MySQL version 5.1.6 and later.

Note: If you use a version of MySQL earlier than version 5.1.6, you willneed to run as the root database user when you run llconfig -i to createand initialize the LoadLeveler configuration tables.

When setting up DB2:

1) EXTSHM=ON is set and exported in the /etc/profile file. Ensure anyprogram that is started without /etc/profile and invokes LoadLevelercommands sets and exports EXTSHM.

2) Ensure that . ~xcatdb/sqllib/db2profile is run for the root ID on theLoadLeveler node where the llconfig -i command will be used toinitialize the database. Substitute the DB2 instance ID you are using, ifthe instance ID is different from xcatdb.

b. Ensure that unixODBC and MySQL Connector/ODBC (the DB2 ODBCdriver is part of the DB2 and no separate install is needed) are installed andset up on all the LoadLeveler nodes that will access the database. Thisincludes setting up /etc/odbcinst.ini, /etc/odbc.ini, and db2cli.ini for DB2 or.odbc.ini in the home directory of root on each node for MySQL. This isrequired for the manager nodes and Schedd nodes.In addition, to ensure database access you must provide a file in the homedirectory of each LoadLeveler administrator. For DB2 copy the db2cli.inifile from the DB instance. For example, if xcatdb is the DB2 instance andloadl is the LoadLeveler administrator:cp ~xcatdb/sqllib/cfg/db2cli.ini ~loadl/db2cli.ini

For MySQL create an .odbc.ini file in the home directory of eachLoadLeveler administrator. Refer to Setting Up MySQL as the xCAT DB or


http://sourceforge.net/apps/mediawiki/xcat/index.php?title=XCAT_Documentation

Setting Up DB2 as the xCAT DB in XCAT Documentation(http://sourceforge.net/apps/mediawiki/xcat/index.php?title=XCAT_Documentation).

5. If you plan to configure LoadLeveler to support the database-basedconfiguration option and want to use the configuration editor, do the followingon the management node:a. Ensure the requirements for the database configuration option are met. See

note 4 on page 14.b. Install the IBM or Apache HTTP Server and xCAT Web interface, as

specified in the Installing the xCAT Web Interface (http://sourceforge.net/apps/mediawiki/xcat/index.php?title=Installing_the_xCAT_Web_Interface).

c. Append LoadLeveler-specific configuration information to the configurationfile for the HTTP Server:1) Enter: cd /usr/IBM/HTTPServer/conf

2) Enter: vi httpd.conf

3) Add the following to the end of the file:Alias /ll-html /var/www/ll/html<Directory /var/www/ll/html>

Order allow,denyAllow from all

</Directory>Alias /ll/ /var/www/ll/cgi-bin/<Location /ll/ >

SetHandler cgi-scriptOptions +ExecCGI

</Location>

d. Ensure the CGI::Session Perl module is available.Download the CGI::Session Perl module at CPAN (http://search.cpan.org/dist/CGI-Session/) and install it using the Makefile.PL section of theinstructions found by following the INSTALL link.

e. After LoadLeveler is installed, ensure the database used by theconfiguration editor is the same as in the /etc/xcat/cfgloc file. The defaultdatabase name for the configuration editor is xcatdb. Update theconfiguration file for the configuration editor if you are using a differentdatabase name than xcatdb:1) Enter: cd /var/www/ll/conf

2) Enter: vi ll.conf

3) Replace xcatdb with the name of the database you are using.

Disk space requirementsCertain directories require specific amounts of disk space.

In order for your installation to be successful, ensure that your directories haveadequate space available.

Table 7 shows disk space requirements for LoadLeveler for Linux facilities.

Table 7. Information about LoadLeveler for Linux facilities

Directory Size

Resource Manager release directory (/opt/ibmll/LoadL/resmgr/full) 80 MB

Scheduler release directory (/opt/ibmll/LoadL/scheduler/full) 85 MB

Local directory 20 MB (minimum)


http://sourceforge.net/apps/mediawiki/xcat/index.php?title=XCAT_Documentation

http://sourceforge.net/apps/mediawiki/xcat/index.php?title=Installing_the_xCAT_Web_Interface

http://search.cpan.org/dist/CGI-Session/

Table 7. Information about LoadLeveler for Linux facilities (continued)

Directory Size

Home directory No limits unless same as release orlocal directory

Release directory for submit-only (/opt/ibmll/LoadL/scheduler/so) 55 MB

Configuration examplesThe following examples show file-based configuration and database-basedconfiguration.

Configuration using files

This example shows a LoadLeveler installation on a small pool of workstationsconnected by a network, using file-based rather than database-based configuration.Many other configurations are possible.

Table 8 demonstrates one way to install LoadLeveler.

Table 8. LoadLeveler configuration example

LoadLeveler element Example directory or location oncentral manager machine

Example directory or location onfirst node in cluster

Machine name mycenman mynode01

Central manager mycenman mycenman

LoadLeveler for Linux homedirectory

/home/loadl (exported) /home/loadl (mounted)

Local directory /var/loadl/mycenman /var/loadl/mynode01

LoadLeveler for Linux releasedirectory1

/opt/ibmll/LoadL/full (exported) /opt/ibmll/LoadL/full (mounted)

Global configuration file LoadL_config (in home directory)

One copy for all machines.

LoadL_config (in home directory)


Local configuration file LoadL_config.local

One copy for each machine.

LoadL_config.local (in localdirectory)

One copy for each machine.

Administration file LoadL_admin (in home directory)


LoadL_admin (in home directory)


spool directory /var/loadl/mycenman/spool (insubdirectory of a local directory)

/var/loadl/mynode01/spool (insubdirectory of a local directory)

execute directory /var/loadl/mycenman/execute (insubdirectory of the local directory)

/var/loadl/mynode01/execute (insubdirectory of the local directory)

log directory /var/loadl/mycenman/log (insubdirectory of the local directory)

/var/loadl/mynode01/log (insubdirectory of the local directory)

1Note: Binaries and manpages from /opt/ibmll/LoadL/scheduler/full are linked to /opt/ibmll/LoadL/full.

Entries in the LoadLeveler for Linux LoadL_admin file and the LoadL_config fileappear in Table 9 on page 17.


Table 9. Sample LoadL_admin and LoadL_config file entries

LoadL_admin file LoadL_config file

mycenman: type=machinecentral_manager=true

mynode01: type=machinecentral_manager=false

RELEASEDIR=/opt/ibmll/LoadL/fullLOCAL_CONFIG=$(tilde)/$(host)/LoadL_config.localLOG=/var/loadl/$(host)/logSPOOL=/var/loadl/$(host)/spoolEXECUTE=/var/loadl/$(host)/execute

Configuration using database

This example shows a LoadLeveler installation on a small pool of workstationsconnected by a network, using database-based configuration. Many otherconfigurations are possible.

Table 10 demonstrates one way to install LoadLeveler.

Table 10. LoadLeveler database-based configuration example

LoadLeveler element Example directory or location oncentral manager machine

Example directory or location onfirst node in cluster

Machine name mycenman mynode01

Central manager mycenman mycenman

LoadLeveler for Linux homedirectory

/home/loadl (exported) /home/loadl (mounted)

Local directory /var/loadl/mycenman /var/loadl/mynode01

LoadLeveler for Linux releasedirectory1

/opt/ibmll/LoadL/full (exported) /opt/ibmll/LoadL/full (mounted)

Master configuration file /etc/LoadL.cfg with LoadLDB = xcatdb /etc/LoadL.cfg with either LoadLDB =xcatdb or LoadLConfigHosts =mycenman, depending on whether itwill access the database or anothernode for the configuration.

spool directory /var/loadl/mycenman/spool (insubdirectory of a local directory)

/var/loadl/mynode01/spool (insubdirectory of a local directory)

execute directory /var/loadl/mycenman/execute (insubdirectory of the local directory)

/var/loadl/mynode01/execute (insubdirectory of the local directory)

log directory /var/loadl/mycenman/log (insubdirectory of the local directory)

/var/loadl/mynode01/log (insubdirectory of the local directory)

1Note: Binaries and manpages from /opt/ibmll/LoadL/scheduler/full are linked to /opt/ibmll/LoadL/full.

Ensure the files odbcinst.ini, odbc.ini, and ~/.odbc.ini for each LoadLeveleradministrator are set up on the nodes that will access the database.

The configuration database is loaded from entries in the LoadLeveler for LinuxLoadL_config file and its corresponding LoadL_admin file by using the commandllconfig -i -f /u/loadl/LoadL_config. These file entries appear in Table 11 on page18.


Table 11. Sample LoadL_admin and LoadL_config file entries for loading database

LoadL_admin file LoadL_config file

mycenman: type=machinemynodes: type=machine_group

machine_list=mynode[01-10]

RELEASEDIR=/opt/ibmll/LoadL/fullLOCAL_ADMIN=$(tilde)/LoadL_adminLOG=/var/loadl/$(host)/logSPOOL=/var/loadl/$(host)/spoolEXECUTE=/var/loadl/$(host)/executeCENTRAL_MANAGER_LIST = mycenman


Chapter 3. Installing LoadLeveler for Linux

You can install LoadLeveler for Linux from the product media.

These installation instructions assume that the LoadLeveler user ID is loadl. See“Create the loadl user ID” for information on using a user ID other than loadl asthe LoadLeveler user ID.

Decide on directories for LoadLevelerDecide on and record the location of your home and local directories.

Use the information in “Installation overview” on page 3 to decide on the locationof the home and local directories. Once you decide, write the locations in the“Actual Directory” column in Table 12.

Table 12. LoadLeveler for Linux directory location checklist

LoadLeveler element Example directory Actual directory

Local directory /var/loadl

Home directory /home/loadl

Release directory /opt/ibmll/LoadL/full

Name of central managermachine

mycenman

Important: The local directory must be unique to each machine in the cluster andmust not be shared by any other machines in the cluster. Also, each machine musthave its own LoadLeveler spool, execute, and log directories.

Log in as rootUnless otherwise indicated, all of the steps performed during LoadLevelerinstallation should be done from the root user ID.

Create the loadl group nameCreate the group name for every node in the LoadLeveler cluster.

This procedure creates the group name only for the node that you are currentlylogged into. It must be repeated for every node in the LoadLeveler cluster. Enterthe command:groupadd -g loadl_group_id loadl

Create the loadl user IDCreate the user ID for every node in the LoadLeveler cluster.

This procedure creates the user ID only for the node that you are currently loggedinto. It must be repeated for every node in the LoadLeveler cluster. Enter thecommand:

© Copyright IBM Corp. 2010, 2012 19

useradd -c loadleveler_user_description -d /home/loadl -s /bin/bash \-u loadl_user_id -g loadl_group_id -m loadl

Note that you must also set a password for the LoadLeveler user ID loadl.

Changing the LoadLeveler user ID

To change the ID you must create a file called /etc/LoadL.cfg and modify this file to pointto the user ID.

If you want to use a different user ID, for example, “joe”, as the LoadLeveler user ID, youmust create a file called /etc/LoadL.cfg and modify this file to point to the user ID. Tocreate this file, follow these steps:

1. su - root

2. Copy the LoadL.cfg file from the samples directory to the /etc directory. For example,issue:

cp /opt/ibmll/LoadL/full/samples/LoadL.cfg /etcchmod 644 /etc/LoadL.cfg

3. Edit the file by entering:

vi /etc/LoadL.cfg

4. Edit the LoadLUserid line to indicate a user ID other than loadl.

5. Edit the LoadLGroupid line to indicate a group associated with the new user ID.

6. For files-based configuration, the default location of the global configuration file is thehome directory of the LoadL user ID.

If necessary, edit the LoadLConfig line to indicate the directory where theLoadL_config file will be located.

7. Enter exit.

Install the LoadLeveler for Linux RPMsUse these steps to install the LoadLeveler for Linux RPMs on the followingplatforms.

Note: If you are running an earlier version of LoadLeveler for Linux, you mustfirst uninstall that version before performing the following steps.

Use the following steps to install the RPMs on:v RHEL 6.2 on LoadLeveler for Linux on x86 Architecture servers and POWER

serversv SLES 11 on LoadLeveler for Linux on x86 Architecture servers1. Put the CD in the drive, then type the following commands and press Enter:

mkdir -p /mnt/cdrommount /dev/cdrom /mnt/cdrom

2. To determine if enough space is available for the LoadLeveler installation in thedefault installation location (/opt/ibmll), type the following command and press<Enter>:df -h /opt

Note: See Table 7 on page 15 for the amount of disk space that the LoadLevelerpackages require.

3. Optional step. To bypass the viewing of and automatically accept theLoadLeveler license:export IBM_LOADL_LICENSE_ACCEPT=yes

4. To install the LoadL-full-license RPM, use the rpm command:


rpm -ivh /mnt/cdrom/LoadL-full-license-Linux-identifier.rpm

5. To accept the license agreement and install the companion "full product" baseRPMs:

Table 13. Procedure for viewing and accepting the license

If you want the option to view and accept thelicense:

The following can only be run ifIBM_LOADL_LICENSE_ACCEPT has been set to yes instep 3. If you want to automatically accept the license andbypass using install_ll:

1. Go to the /opt/ibmll/LoadL/sbin directory:

cd /opt/ibmll/LoadL/sbin

2. Type the following installation command andpress Enter:

./install_ll -d rpmpath

where:

-d rpmpathIs an optional flag specifying thedirectory that contains the LoadLevelerpackages. To use the default directory(/mnt/cdrom), you do not need to specifythis flag.

Note: You can install just the LoadL-scheduler orLoadL-resmgr component by using the followingcommand:

./install_ll -d rpmpath -c component

where:

-c component is either -c resmgr or -c scheduler.

If you install just the LoadL-scheduler orLoadL-resmgr component, ensure that the pathsfor LoadLeveler binaries in your configuration arecorrect based on the components you haveinstalled. See Table 14 on page 25 for moreinformation.

Install the LoadLeveler full-product RPMs using the rpmcommand:

rpm -ihv LoadL-resmgr-full-Linux_identifier.rpmrpm -ihv LoadL-scheduler-full-Linux_identifier.rpm

Notes:

a. The licenses provided by the LoadL-full-license RPM areLoadL-resmgr-license-5.1 and LoadL-scheduler-license-5.1.

b. The install_ll script runs the License Acceptance Program (LAP). If thelicense agreement is accepted, it installs the RPMs for LoadL-resmgr-fulland LoadL-scheduler-full for the platform.

c. If you decide not to accept the license agreement, the -e option of the rpmcommand can be used to uninstall the LoadL-full-license RPM. Forexample, to uninstall the RPM, LoadL-full-license-RH6-X86_64-5.1.0.3-0.x86_64.rpm, issue the following command and press Enter:rpm -e LoadL-full-license-RH6-X86_64-5.1.0.3-0

6. To install the optional LoadL-utils RPM, use the rpm command:rpm -ihv /mnt/cdrom/LoadL-utils-Linux-identifier.rpm

7. To install the LoadLeveler service updates for 5.1 corresponding to the RPMsyou just installed:

Chapter 3. Installing LoadLeveler for Linux 21

a. Download Linux service updates by accessing Fix Central(http://www-933.ibm.com/support/fixcentral/?productGroup0=ibm/fcpower&productGroup1=ibm/ClusterSoftware&productGroup2=ibm/power/IBM).

b. After downloading, select the “Installation Instructions” tab for instructionson applying the update.

8. For Blue Gene, download the additional LoadLeveler RPM,LoadL-scheduler-bluegene, from Fix Central (http://www-933.ibm.com/support/fixcentral/?productGroup0=ibm/fcpower&productGroup1=ibm/ClusterSoftware&productGroup2=ibm/power/IBM).Install the LoadLeveler Blue Gene RPM using this command:rpm -ihv LoadL-scheduler-bluegene-RH6-PPC64-5.1.0.3-0.ppc64.rpm [--nodeps]

Notes:

a. The --nodeps option might be required when installing on Blue Genesystems that are mounting the Blue Gene software stack.

b. In order for LoadLeveler to operate correctly in the Blue Gene/Qenvironment, the LoadLeveler plugin must be specified as the plugin forIBM runjob.mux. This can be done by adding the plugin under therunjob.mux section in the bg.properties file:[runjob.mux]...plugin = /usr/lib64/libllrunjob_mux.so

# Fully qualified path to the plugin used for communicating with a job scheduler.# This value can be updated by the runjob_mux_refresh_config command on the# Login Node where a runjob_mux process runs.

...

When updates are applied for the LoadLeveler software, you should refreshthe runjob.mux by using the runjob_mux_refresh_config command.

Run the LoadLeveler initialization script llinit (optional)This step can be run for each of the machines in the LoadLeveler cluster to set upconfiguration files, spool, log and execute directories, and symbolic links to theLoadLeveler user ID's home directory.

If you skip this step, ensure that the LoadLeveler spool execute, and log directorieshave been created in the local directory with permissions set to 700, 1777, and 775,respectively, and that either file-based or database-based configuration has been setup.1. You must perform this step using the LoadLeveler user ID. These instructions

use loadl as this ID. To switch to the loadl ID, enter the following:su - loadl

2. Ensure that your HOME environment variable is set to loadl's home directory.3. Change the current directory to the bin subdirectory in the release directory by

entering:cd /opt/ibmll/LoadL/scheduler/full/bin

4. To run llinit, ensure that you have write privileges in the LoadLeveler home,local, and /tmp directories.

5. Enter the llinit command.For example, to run the llinit command with a local directory of /var/loadl, arelease directory of /opt/ibmll/LoadL/full, and a central manager machine withthe host name mycenman enter the following:./llinit -local /var/loadl -release /opt/ibmll/LoadL/full -cm mycenman




Note: Ensure that the local directory exists before running the precedingcommands.

About llinit

llinit does the following:v Copies the LoadL_admin and the LoadL_config files from the release directory

(in the samples subdirectory) into the home directory of loadl.

Note: These files are a common resource for all of the machines in theLoadLeveler cluster and, therefore, must be made accessible to all members ofthe LoadLeveler pool.

v Creates the LoadLeveler spool, execute, and log directories in the local directorywith permissions set to 700, 1777, and 775, respectively.

v Copies the LoadL_config.local file from the release directory (in the samplessubdirectory) into the local directory.

v Creates symbolic links from the loadl home directory to the spool, execute, andlog subdirectories and the LoadL_config.local file in the local directory (if homeand local directories are not identical).

v Creates symbolic links from the home directory to the bin, man, samples, andinclude subdirectories in the release directory.

The preceding files are copied and directories and symbolic links are created onlyif they do not already exist.

Note: You can change the locations of the spool, execute, and log directories bychanging the associated paths in the configuration keywords spool, execute, andlog. For example, if you want to move the spool, execute, and log directory from/var/loadl into /tmp/loadl, with appropriate permissions set, you can do so but youmust create /tmp/loadl/spool, /tmp/loadl/execute, and /tmp/loadl/log in /tmp/loadlor LoadLeveler will not start up.

See LoadLeveler: Command and API Reference for more information about the llinitcommand.

Make the LoadLeveler for Linux man pages available (optional)If you do not want to make the man pages available to users, you can skip thisstep.

Add the following path to your MANPATH:/opt/ibmll/LoadL/full/man:/opt/ibmll/LoadL/resmgr/full/man

Set the default level for the llstatus and llrstatus commands (optional)If you do not want to change the default level for llstatus and llrstatus, you canskip this step. The default level is cluster. LoadLeveler versions prior to version 5.1displayed machine level output in llstatus and llrstatus. To continue to see thatoutput format by default, export LOADL_STATUS_LEVEL=machine in theenvironment.

If you want to change the default level, add one of the following to your profile:export LOADL_STATUS_LEVEL=machine


so the command output is equivalent to specifying with -L machine, or:export LOADL_STATUS_LEVEL=machine_group

so the command output is equivalent to specifying with -L machine_group.

Repeat the appropriate steps for each LoadLeveler for Linux machineRepeat only the necessary steps.

If the release directory can be mounted with AFS or NFS, you can mount thedirectory, skip “Install the LoadLeveler for Linux RPMs” on page 20 and continuewith “Run the LoadLeveler initialization script llinit (optional)” on page 22 forother machines in your configuration.

On the machines that you do not install the RPMs:v You must create the symbolic links for the shared libraries in the installed

components.– If installing 64-bit LoadLeveler for Linux on x86 Architecture and IBM Power

Systems servers, as root enter the following commands:# Symbolic links for resmgr componentln -sf /opt/ibmll/LoadL/resmgr/full/lib/libllrapi.so /usr/lib64/libllrapi.so.1.0.0ln -sf /usr/lib64/libllrapi.so.1.0.0 /usr/lib64/libllrapi.so.1ln -sf /usr/lib64/libllrapi.so.1 /usr/lib64/libllrapi.soln -sf /opt/ibmll/LoadL/resmgr/full/lib/libllrdbapi.so /usr/lib64/libllrdbapi.so.1.0.0ln -sf /usr/lib64/libllrdbapi.so.1.0.0 /usr/lib64/libllrdbapi.so.1ln -sf /usr/lib64/libllrdbapi.so.1 /usr/lib64/libllrdbapi.soln -sf /opt/ibmll/LoadL/resmgr/full/lib/libpermapi.so /usr/lib64/libpermapi.so.1.0.0ln -sf /usr/lib64/libpermapi.so.1.0.0 /usr/lib64/libpermapi.so.1ln -sf /usr/lib64/libpermapi.so.1 /usr/lib64/libpermapi.so

# Symbolic links for scheduler componentln -sf /opt/ibmll/LoadL/scheduler/full/lib/libllapi.so /usr/lib64/libllapi.so.1.0.0ln -sf /usr/lib64/libllapi.so.1.0.0 /usr/lib64/libllapi.so.1ln -sf /usr/lib64/libllapi.so.1 /usr/lib64/libllapi.soln -sf /opt/ibmll/LoadL/scheduler/full/lib/liblldbapi.so /usr/lib64/liblldbapi.so.1.0.0ln -sf /usr/lib64/liblldbapi.so.1.0.0 /usr/lib64/liblldbapi.so.1ln -sf /usr/lib64/liblldbapi.so.1 /usr/lib64/liblldbapi.so

– On RHEL 6.2 for IBM Power Systems, if Blue Gene support is required and ifthe LoadL-scheduler-bluegene-RH6-PPC64-5.1.0.x-0.ppc64.rpm has beeninstalled on a shared file system, you must create the symbolic links for thelibraries. As root enter the following commands:ln -s /opt/ibmll/LoadL/scheduler/full/lib64/libllbgq.so /usr/lib64/libllbgq.so.1.0.0ln -s /usr/lib64/libllbgq.so.1.0.0 /usr/lib64/libllbgq.so.1ln -s /usr/lib64/libllbgq.so.1 /usr/lib64/libllbgq.soln -s /opt/ibmll/LoadL/scheduler/full/lib64/libllrunjob_mux.so \

/usr/lib64/libllrunjob_mux.so.1.0.0ln -s /usr/lib64/libllrunjob_mux.so.1.0.0 /usr/lib64/libllrunjob_mux.so.1ln -s /usr/lib64/libllrunjob_mux.so.1 /usr/lib64/libllrunjob_mux.so

v Run ldconfig after creating the symbolic links:/sbin/ldconfig -X

v Create the symbolic links to /usr/bin/LoadL_configurator:If only the scheduler component is installed or if both scheduler and resmgr areinstalled, enter:ln -s /opt/ibmll/LoadL/scheduler/full/bin/LoadL_configurator /usr/bin/LoadL_configurator

If only resmgr is installed, enter:ls -s /opt/ibmll/LoadL/resmgr/full/bin/LoadL_configurator /usr/bin/LoadL_configurator

v You must create the symbolic link for the LoadLeveler message catalog. If thedirectory /usr/share/locale/C/LC_MESSAGES does not exist, it must first becreated. Then, as root, enter the commands.


cd /usr/share/locale/C/LC_MESSAGESln -s /opt/ibmll/LoadL/full/msg/loadl.cat loadl.cat

Configure your LoadLeveler for Linux systemThis consists of editing certain files to meet the needs of your installation.

Setting up your LoadLeveler system for file-based configuration consists of editingthe LoadL_config and LoadL_admin files and the LoadL_config.local local file tomeet the needs of your installation.

Setting up your LoadLeveler system for database-based configuration consists ofediting the master configuration file to specify an ODBC data source name (DSN)that is configured so that the specified database user can access the xCAT databaseschema; editing a global configuration file with initial settings to meet the needs ofyour installation such as central_manager_list, resource_mgr_list, andloadl_admin; editing a LoadL_admin file with machine information; and theninitializing the database using the llconfig -i command and either pointing to ordefaulting to the global configuration file. Then use llconfig -c with the -h optionto make any local or machine-specific changes that are needed. For moreinformation about the llconfig command, see LoadLeveler: Command and APIReference.

Note: To verify that the ODBC DSN stanza has been configured correctly, run thefollowing command:/usr/local/bin/isql -v <dsn_stanza_name>

If a connection cannot be made to the database, check that access privileges havebeen granted in the database for the node you are running on, fix any errors in theodbc configuration files (/etc/odbc.ini, /etc/odbcinst.ini, and ~/.odbc.ini ordb2cli.ini), and rerun the command.

Table 14. Configuration considerations when the scheduler component is not installed on allmachines

If the scheduler component is not installed on all machines in the cluster, then all of theLoadLeveler binaries will not be available in the same directory. The appropriate keywordsin the LoadLeveler configuration must be set so that the binaries for the resource managerand scheduler daemons can be located.

The NEGOTIATOR keyword must specify the location of the LoadL_negotiator file in thescheduler installation directory on the machines where the scheduler component isinstalled. If the default account validation program is being used, then theACCT_VALIDATION keyword must specify the location of the llacctval file in thescheduler installation directory where the scheduler component is installed. The BINkeyword must be set to the installation directory for the resource manager binaries and theconfiguration keywords for all LoadLeveler daemons other than the NEGOTIATOR mustuse the resource manager bin directory. For example:

BIN = /opt/ibmll/LoadL/resmgr/full/bin

NEGOTIATOR = /opt/ibmll/LoadL/scheduler/full/bin/LoadL_negotiatorACCT_VALIDATION = /opt/ibmll/LoadL/scheduler/full/bin/llacctval

SCHEDD = $(BIN)/LoadL_scheddSTARTD = $(BIN)/LoadL_startdSTARTER = $(BIN)/LoadL_starterMASTER = $(BIN)/LoadL_masterRESOURCE_MGR =$(BIN)/LoadL_resource_mgrREGION_MGR = $(BIN)/LoadL_region_mgr


For detailed instructions on configuring the LoadLeveler environment, seeLoadLeveler: Using and Administering.

Once you configure your system, you can verify your configuration by running thecommand llctl ckconfig. See post-installation considerations in LoadLeveler: Usingand Administering for more information.


Chapter 4. Installing submit-only LoadLeveler for Linux

Use this procedure to install submit-only LoadLeveler machines for Linux.

This topic also explains how to set up LoadLeveler for Linux submit-onlymachines.

Submit-only LoadLeveler for Linux planning considerationsMake the following considerations when planning submit-only LoadLeveler forLinux.

Before installing submit-only LoadLeveler, make sure you have read the followingtopics:v “Configuration planning” on page 4v “Migration considerations” on page 7v “Hardware, software, and disk requirements” on page 11

For file-based configuration, if the release directory will be shared amongsubmit-only machines using NFS, you must either allow all files to be worldreadable (by default), or the file system must be exported to all hosts that will berunning submit-only LoadLeveler.

Submit-only LoadLeveler for Linux disk space requirementsCertain directories require specific amounts of disk space.

In order for your installation to be successful, ensure that your directories haveadequate space available. Table 15 shows disk space requirements for submit-onlyLoadLeveler for Linux facilities.

Table 15. Submit-only LoadLeveler for Linux disk space requirements

Platform Directory Size

LoadLeveler forLinux on x86Architecture andLoadLeveler forLinux on POWER

Release directory ((/opt/ibmll/LoadL/scheduler/so)

55 MB

Set up the submit-only LoadLeveler user ID and groupUse this procedure to set up the submit-only LoadLeveler for Linux user ID andgroup.

This step must be done as root.1. Create a user ID named loadl and a group named loadl on the submit-only

machine. You must use the same user ID number for all LoadLeveler machines.The same is true for the group ID. Include the loadl user ID in the loadl group.

2. Create a home directory for the loadl user ID as shown in Table 16 on page 28.

Notes:

1. The owner and group of all directories created must be loadl.


2. The release directory can be shared among submit-only machines.3. There are no requirements that each directory be in a different area. An

example configuration is:

Table 16. Submit-only LoadLeveler for Linux directory information

Directory Description

/home/loadl Home directory

/opt/ibmll/LoadL/scheduler/so Release directory

Install the submit-only LoadLeveler for Linux RPMsUse this procedure to install the submit-only LoadLeveler for Linux RPMs on thefollowing platform:v RHEL 6.2 and SLES 11 on LoadLeveler for Linux on x86 Architecture serversv RHEL 6.2 on IBM Power Systems servers1. Log in as root.2. Put the CD in the drive, then type the following command and press Enter:

mkdir -p /mnt/cdrommount /dev/cdrom /mnt/cdrom

3. To determine the amount of space available for the LoadLeveler installation inthe default installation location (/opt/ibmll), type the following command andpress Enter:df -h /opt

Note: See Table 7 on page 15 for the amount of disk space that the LoadLevelerpackages require.

4. To install the LoadL-so-license RPM, use the rpm command:rpm -ivh /mnt/cdrom/LoadL-so-Linux-identifier.rpm

5. To install the LoadLeveler service updates for 5.1 corresponding to the baseRPM you just installed:a. Download Linux service updates by accessing Fix Central

(http://www-933.ibm.com/support/fixcentral/?productGroup0=ibm/fcpower&productGroup1=ibm/ClusterSoftware&productGroup2=ibm/power/IBM).

b. After downloading, select the “Installation Instructions” tab for instructionson applying the update.

Set up your submit-only LoadLeveler configurationPerform this step as loadl. Ensure that your HOME environment variable is set toloadl's home directory.1. Add /opt/ibmll/LoadL/scheduler/so/bin to the PATH environment variable.2. Add /opt/ibmll/LoadL/scheduler/so/man to your MANPATH.

If you installed a LoadL.msg.lang fileset, add the following preceding/opt/ibmll/LoadL/scheduler/so/man:/opt/ibmll/LoadL/scheduler/so/man/$LANG

3. Set the LOADL_STATUS_LEVEL environment variable to machine ormachine_group, if you want to change the default level of llstatus and llrstatusoutput, which is cluster.



Update the submit-only LoadLeveler cluster configurationYou must update the configuration for your cluster for submit-only machines.

Updating File-based configuration

For files-based configuration, it is suggested that the LoadL_admin andLoadL_config files be the same on all machines, with local differences made in theLoadL_config.local file. The files copied from the submit-only samples directorycontain the few, necessary entries for submit-only LoadLeveler. However, you canmake the LoadL_admin and LoadL_config files the same as those used by aregular LoadLeveler client.

To use the files from the submit-only samples, copy and rename the three filesfrom the samples directory to the loadl home directory and the local directory.This step assumes the local directory is $HOME/hostname.cp /opt/ibmll/LoadL/scheduler/so/samples/LoadL_admin $HOMEcp /opt/ibmll/LoadL/scheduler/so/samples/LoadL_config.so $HOME/LoadL_configcp /opt/ibmll/LoadL/scheduler/so/samples/LoadL_config.local.so \

$HOME/hostname/LoadL_config.local

To update a file-based configuration for the submit-only function, perform thefollowing steps:1. The LoadL_admin file must have one or more machines defined as public

schedulers (schedd_host = true).a. If the LoadL_admin file is not shared by the regular LoadLeveler pool

machines, edit the machine stanza of this LoadL_admin file and specify thenames of any public schedulers, as defined in the LoadL_admin file usedby the regular LoadLeveler pool. For example:llhost1: type = machineschedd_host = truellhost2: type = machineschedd_host = true

b. Define submit-only machines stanzas to the LoadL_admin file if you intendthe submit-only machine to submit jobs to a cluster that setsMACHINE_AUTHENTICATE = TRUE. If MACHINE_AUTHENTICATE =TRUE is set, the associated machines will accept jobs only from machinesthat are defined in the LoadL_admin fileEdit the machine stanza of the LoadL_admin file used by the LoadLevelerpool machines, and add the name of each submit-only machine, specifyingthe following keywords:so_ll3: type = machineschedd_host = falsesubmit_only = trueso_ll4: type = machineschedd_host = falsesubmit_only = true

2. In the LoadL_config.local file on the submit-only machine, specify thefollowing:SCHEDD_RUNS_HERE = FALSESTARTD_RUNS_HERE = FALSESTART_DAEMONS = FALSE

If the configuration file specifies account validation (ACCT=A_VALIDATE), thefollowing keyword needs to be added:ACCT_VALIDATION=/opt/ibmll/LoadL/scheduler/so/bin/llacctval

Chapter 4. Installing submit-only LoadLeveler for Linux 29

See the LoadL_config.so file in /opt/ibmll/LoadL/scheduler/so/samples foradditional keywords you can use.

Updating Database-based configuration

To update a database-based configuration for submit-only, perform the followingsteps:1. As root, create a master configuration file, /etc/LoadL.cfg, on each submit-only

node with LoadLConfigHosts set to designate a machine or list of machinesthat will provide the configuration to the submit-only node:LoadLConfigHosts = my_schedd my_cenman

where my_schedd and my_cenman are nodes that are set up to access thedatabase.

Note: For steps 2 and 3, you can make these changes to the configuration filesused to initialize the configuration database before running llconfig -i; or afterthe database has been initialized, you can update the values using the changeoption of llconfig or the configuration editor.

2. As a LoadLeveler administrator, define one or more schedd machines as publicschedulers using the schedd_host = true keyword.

3. Define submit-only machine stanzas in a stanza file, such assubmit_only_admin if you intend the submit-only machine to submit jobs to acluster that sets MACHINE_AUTHENTICATE = TRUE. IfMACHINE_AUTHENTICATE = TRUE is set, the associated machines willaccept jobs only from machines that are defined for the cluster.v You can define each submit-only machine separately:

so_ll3: type = machinesubmit_only = trueschedd_runs_here = falsestartd_runs_here = false

so_ll4: type = machinesubmit_only = trueschedd_runs_here = falsestartd_runs_here = false

v Or you can define it as part of a submit-only machine group:submit_only: type = machine_group

machine_list = so_11[3-4]submit_only = trueschedd_runs_here = falsestartd_runs_here = false

Use the commands:llconfig -s -a -f /u/loadl/submit_only_adminllconfig -h so_ll3 so_ll4 -c start_daemons=false

to add the submit-only machines to the cluster.

From the submit-only machine, you can now submit jobs, query job status, andcancel jobs with the following commands: llsubmit, llq, and llcancel.

Install submit-only LoadLeveler for Linux on other machines in thepool

Repeat the previous steps for each machine in the pool.


If the release directory can be mounted using NFS, you do not have to receive theinstallation image; you can mount the release directory onto the machine.However, if you do this, you must create symbolic links for the shared library asdescribed in “Repeat the appropriate steps for each LoadLeveler for Linuxmachine” on page 24.

Running reservation commands without starting LoadLevelerThe reservation commands are located in the LoadLeveler scheduler fileset,LoadL.scheduler.full. You can run the reservation commands on a node withoutstarting LoadLeveler daemons. Configuring a node similarly to a submit-only nodesetup will allow reservation commands to be used for xCAT rolling updateswithout starting LoadLeveler on the node.

To run the reservation commands on a node without starting LoadLeveler, followthese steps:1. Install the LoadL.scheduler.full fileset.2. Set up the master configuration file, /etc/LoadL.cfg, on the node to point to

where the configuration data is located.3. Set up the node to be able to run reservation, by setting

reservation_permitted=true in its machine stanza. Also definemax_reservations for the users who can make reservations.

4. Set up the node to act like a “submit-only” node by setting these keywords:SCHEDD_RUNS_HERE = false and STARTD_RUNS_HERE = false andSTART_DAEMONS = false. These settings will prevent the node fromaccidentally starting.

5. In the configuration, set up another node that runs the Schedd withSCHEDD_HOST = true.

6. Start the rest of the nodes in the cluster.

Chapter 4. Installing submit-only LoadLeveler for Linux 31

Accessibility features for LoadLeveler

Accessibility features help users who have a disability, such as restricted mobilityor limited vision, to use information technology products successfully.

Accessibility featuresThe following list includes the major accessibility features in IBM LoadLeveler:v Keyboard-only operationv Interfaces that are commonly used by screen readersv Keys that are discernible by touch but do not activate just by touching themv Industry-standard devices for ports and connectorsv The attachment of alternative input and output devices

The IBM Cluster information center, and its related publications, areaccessibility-enabled. The accessibility features of the information center aredescribed in the IBM Cluster information center (http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/topic/com.ibm.cluster.addinfo.doc/access.html).

Keyboard navigationThis product uses standard Microsoft Windows navigation keys.

IBM and accessibilitySee the IBM Human Ability and Accessibility Center (http://www.ibm.com/able/)for more information about the commitment that IBM has to accessibility.



http://www.ibm.com/able/

Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document inother countries. Consult your local IBM representative for information on theproducts and services currently available in your area. Any reference to an IBMproduct, program, or service is not intended to state or imply that only IBM'sproduct, program, or service may be used. Any functionally equivalent product,program, or service that does not infringe any of IBM's intellectual property rightsmay be used instead. However, it is the user's responsibility to evaluate and verifythe operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matterdescribed in this document. The furnishing of this document does not give youany license to these patents. You can send license inquiries, in writing, to:

IBM Director of LicensingIBM CorporationNorth Castle DriveArmonk, NY 10504-1785USA

For license inquiries regarding double-byte character set (DBCS) information,contact the IBM Intellectual Property Department in your country or sendinquiries, in writing, to:

Intellectual Property LicensingLegal and Intellectual Property LawIBM Japan Ltd.1623-14, Shimotsuruma, Yamato-shiKanagawa 242-8502 Japan

The following paragraph does not apply to the United Kingdom or any othercountry where such provisions are inconsistent with local law:

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THISPUBLICATION “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHEREXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIEDWARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESSFOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express orimplied warranties in certain transactions, therefore, this statement may not applyto you.

This information could include technical inaccuracies or typographical errors.Changes are periodically made to the information herein; these changes will beincorporated in new editions of the publication. IBM may make improvementsand/or changes in the product(s) and/or the program(s) described in thispublication at any time without notice.

Any references in this information to non-IBM Web sites are provided forconvenience only and do not in any manner serve as an endorsement of those Web


sites. The materials at those Web sites are not part of the materials for this IBMproduct and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way itbelieves appropriate without incurring any obligation to you.

Licensees of this program who wish to have information about it for the purposeof enabling: (i) the exchange of information between independently createdprograms and other programs (including this one) and (ii) the mutual use of theinformation which has been exchanged, should contact:

IBM CorporationIntellectual Property LawMail Station P3002455 South Road,Poughkeepsie, NY 12601-5400USA

Such information may be available, subject to appropriate terms and conditions,including in some cases, payment or a fee.

The licensed program described in this document and all licensed materialavailable for it are provided by IBM under terms of the IBM Customer Agreement,IBM International Program License Agreement or any equivalent agreementbetween us.

This information contains examples of data and reports used in daily businessoperations. To illustrate them as completely as possible, the examples include thenames of individuals, companies, brands, and products. All of these names arefictitious and any similarity to the names and addresses used by an actual businessenterprise is entirely coincidental.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, whichillustrate programming techniques on various operating platforms. You may copy,modify, and distribute these sample programs in any form without payment toIBM, for the purposes of developing, using, marketing or distributing applicationprograms conforming to the application programming interface for the operatingplatform for which the sample programs are written. These examples have notbeen thoroughly tested under all conditions. IBM, therefore, cannot guarantee orimply reliability, serviceability, or function of these programs. The sampleprograms are provided "AS IS", without warranty of any kind. IBM shall not beliable for any damages arising out of your use of the sample programs.

Each copy or any portion of these sample programs or any derivative work, mustinclude a copyright notice as follows:

© your company name) (year). Portions of this code are derived from IBM Corp.Sample Programs.

© Copyright IBM Corp. _enter the year or years.

If you are viewing this information softcopy, the photographs and colorillustrations may not appear.


Trademarks

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks ofInternational Business Machines Corp., registered in many jurisdictions worldwide.Other product and service names might be trademarks of IBM or other companies.A current list of IBM trademarks is available on the Web at Copyright andtrademark information (http://www.ibm.com/legal/copytrade.shtml).

Intel, Intel Inside (logos), MMX and Pentium are trademarks of Intel Corporationin the United States, other countries, or both.

Java and all Java-based trademarks and logos are trademarks or registeredtrademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Linux is a registered trademark of Linus Torvalds in the United States, othercountries, or both.

Microsoft and Windows are trademarks of Microsoft Corporation in the UnitedStates, other countries, or both.

Red Hat, the Red Hat “Shadow Man” logo, and all Red Hat-based trademarks andlogos are trademarks or registered trademarks of Red Hat, Inc., in the UnitedStates and other countries.

UNIX is a registered trademark of the Open Group in the United States and othercountries.

Other company, product, or service names may be trademarks or service marks ofothers.

Notices 37

http://www.ibm.com/legal/copytrade.shtml

http://www.ibm.com/legal/copytrade.shtml

Glossary

This glossary includes terms and definitions forLoadLeveler.

The following cross-references are used in thisglossary:v See refers you from a term to a preferred

synonym, or from an acronym or abbreviationto the defined full form.

v See also refers you to a related or contrastingterm.

To view glossaries for other IBM products, go toIBM Terminology (http://www-01.ibm.com/software/globalization/terminology/index.jsp).

A

AFS A distributed file system for largenetworks that is known for its ease ofadministration and expandability.

AIX A UNIX operating system developed byIBM that is designed and optimized torun on POWER microprocessor-basedhardware such as servers, workstations,and blades.

authenticationThe process of validating the identity of auser or server.

authorizationThe process of obtaining permission toperform specific actions.

B

Berkeley Load AverageThe average number of processes on theoperating system's ready-to-run queue.

C

C languageA language used to develop applicationprograms in compact, efficient code thatcan be run on different types ofcomputers with minimal change.

client A system or process that is dependent onanother system or process (usually calledthe server) to provide it with access todata, services, programs, or resources.

clusterA collection of complete systems thatwork together to provide a single, unifiedcomputing capability.

D

daemonA program that runs unattended toperform continuous or periodic functions,such as network control.

DCE See Distributed Computing Environment.

defaultPertaining to an attribute, value, or optionthat is assumed when none is explicitlyspecified.

DFS See Distributed File System.

Distributed Computing Environment (DCE)In network computing, a set of servicesand tools that supports the creation, use,and maintenance of distributedapplications across heterogeneousoperating systems and networks.

Distributed File Service (DFS)A component of a Distributed ComputingEnvironment (DCE) that enables a single,integrated file system to be shared amongall DCE users and host computers in aDCE cell. DFS prevents DCE users fromsimultaneously modifying the sameinformation.

F

flexible jobA job step that is only used forscheduling resources for a flexiblereservation.

flexible reservationA reservation that starts as soon asresources are first available.

H

host A computer that is connected to anetwork and provides an access point tothat network. The host can be a client, aserver, or both a client and serversimultaneously.


http://www-01.ibm.com/software/globalization/terminology/index.jsp

L

LAPI See low-level application programminginterface.

low-level application programming interface(LAPI)

An IBM message-passing interface thatimplements a one-sided communicationmodel.

M

MCM See multiple chip module.

memory affinityA feature available in AIX to allocatememory attached to the same multiplechip module (MCM) on which the processruns. Memory affinity improves theperformance of applications on IBMSystem p® servers.

menu A displayed list of items from which auser can make a selection.

Message Passing Interface (MPI)A library specification for messagepassing. MPI is a standard applicationprogramming interface (API) that can beused with parallel applications and thatuses the best features of a number ofexisting message-passing systems.

MPI See Message Passing Interface.

MPICH2A portable implementation of theMessage Passing Interface (MPI).

multiple chip module (MCM)The fundamental, processor, buildingblock of IBM System p servers.

N

networkIn data communication, a configuration inwhich two or more locations arephysically connected for the purpose ofexchanging data.

Network File System (NFS)A protocol, developed by SunMicrosystems, Incorporated, that enablesa computer to access files over a networkas if they were on its local disks.

NFS See Network File System.

node A computer location defined in anetwork.

P

parameterA value or reference passed to a function,command, or program that serves asinput or controls actions. The value issupplied by a user or by another programor process.

processA separately executable unit of work.

R

RDMASee Remote Direct Memory Access.

Remote Direct Memory Access (RDMA)A communication technique in which datais transmitted from the memory of onecomputer to that of another withoutpassing through a processor. RDMAaccommodates increased network speeds.

resource set (RSet)A data structure in AIX used to representphysical resources such as processors andmemory. AIX uses resource sets to restricta set of processes to a subset of thesystem's physical resources.

RSet See resource set.

S

server In a network, hardware or software thatprovides facilities to clients. Examples ofa server are a file server, a printer server,or a mail server.

shell A software interface between users and anoperating system. Shells generally fall intoone of two categories: a command lineshell, which provides a command lineinterface to the operating system; and agraphical shell, which provides agraphical user interface (GUI).

simultaneous multithreading (SMT)Pertaining to a processor design thatcombines hardware multithreading withsuperscalar processor technology. UsingSMT, a single physical processor emulatesmultiple processors by enabling multiplethreads to issue instructionssimultaneously during each cycle.

SMT See simultaneous multithreading.


system administratorThe person who controls and manages acomputer system.

T

TCP See Transmission Control Protocol.

Transmission Control Protocol (TCP)A communication protocol used in theInternet and in any network that followsthe Internet Engineering Task Force (IETF)standards for internetwork protocol. TCPprovides a reliable host-to-host protocol inpacket-switched communication networksand in interconnected systems of suchnetworks.

U

UDP See User Datagram Protocol.

User Datagram Protocol (UDP)An Internet protocol that providesunreliable, connectionless datagramservice. It enables an application programon one machine or process to send adatagram to an application program onanother machine or process.

W

workflowAn application that has been partitionedinto a complex sequence ofinterdependent jobs is called a workflow.The execution of jobs in a workflow maydepend on the success/failure ofpreviously executed jobs. Some jobs in aworkflow may need to use output of jobsthat executed before them; jobs that arenot interdependent may be able toexecute in parallel.

workflow engineA software tool that manages theexecution of a workflow. By integratingwith LoadLeveler, a workflow engine canbenefit from features like flexiblereservations and state change notificationsto automate several workflows in parallel.

working directoryThe active directory. When a file name isspecified without a directory, the currentdirectory is searched.

workstationA configuration of input/outputequipment at which an operator works. A

workstation is a terminal ormicrocomputer at which a user can runapplications and that is usually connectedto a mainframe or a network.

Glossary 41

Index

Aaccessibility 33

keyboard 33shortcut keys 33

Ccommands

running llinit on LoadLeveler for AIX 23configuration

examples 16planning 4

configuration using databaseexample 16

configuration using filesexample 16

configuringLoadLeveler for Linux 25submit-only LoadLeveler 28

considerationsfor upgrading daemons and nodes 9

conventions and terminology viicreating loadl group name

submit-only LoadLeveler 27creating loadl user ID 19

submit-only LoadLeveler 27creating the loadl group name 19

Ddaemon failover and shutdown

procedures 9directories

LoadLeveler 19disability 33disk space

requirements 15requirements for submit-only LoadLeveler for Linux 27

diskless environmentinstalling LoadLeveler 6

Ggroup name

creating for submit-only LoadLeveler 27loadl, creating 19

Hhardware requirements

LoadLeveler for Linux on POWER 11LoadLeveler for Linux on x86 Architecture 11

Iinitialization script

running llinit on LoadLeveler 22

installationconfiguration planning 4considerations 6overview 3running llinit initialization script on LoadLeveler 22

installingLoadLeveler for Linux 19LoadLeveler for Linux RPMs 20submit-only LoadLeveler for Linux on other machines in

the pool 31submit-only LoadLeveler for Linux RPMs 28

installing LoadLevelerin a diskless environment 6

installing submit-only LoadLeveler for Linux 27introduction, LoadLeveler 1

Llicense inquiries 35links, symbolic

on LoadLeveler for Linux 24llinit command

running on LoadLeveler for AIX 23llinit initialization script

running on LoadLeveler 22llstatus and llrstatus commands

setting the default level 23loadl group name

creating 19creating for submit-only LoadLeveler 27

loadl user IDcreating 19creating for submit-only LoadLeveler 27

LoadLevelerdirectories 19running llinit initialization script 22

LoadLeveler for AIXllinit command 23

LoadLeveler for Linuxconfiguring 25creating symbolic links 24installing 19installing the RPMs 20man pages 23RPMs 3

LoadLeveler introduction 1

Mman pages

using on LoadLeveler for Linux 23migration considerations 7

Nnotices 35

© Copyright IBM Corp. 2010, 2012 43

Ooverview

installation 3

Ppatent information 35planning

configuration 4for LoadLeveler for Linux RPMs 3for submit-only LoadLeveler for Linux 27general considerations 6pre-installation considerations 3

pre-installation planningconfiguration 4for LoadLeveler for Linux RPMs 3general considerations 6installation overview 3

proceduresdaemon failover and shutdown 9

Rreservation commands

running, without starting Loadleveler daemons 31RPMs

installing for LoadLeveler for Linux 20installing for submit-only LoadLeveler for Linux 28LoadLeveler for Linux 3

running reservation commands without startingLoadleveler 31

Ssetting the default level

llstatus and llrstatus commands 23shortcut keys

keyboard 33software requirements

LoadLeveler for Linux 12submit-only LoadLeveler

configuring 28creating loadl group name 27creating loadl user ID 27

submit-only LoadLeveler for Linuxdisk space requirements 27installing 27installing on other machines in the pool 31installing the RPMs 28planning considerations 27updating cluster configuration 29

symbolic linkson LoadLeveler for Linux 24

Ttrademarks 37

Uupdating

submit-only LoadLeveler for Linux clusterconfiguration 29

upgradingdaemons and nodes 9

using databaseconfiguration example 16

using filesconfiguration example 16


��

Product Number: 5725-G015641-LL15641-LL3

Printed in USA

SC23-6789-00

ibm loadleveler for linux -...

Documents