3.2.9 Dynamic DeploymentDynamic deployment is a mechanism that allows rapid, automated installation and start up of HTCondorresources on a given machine. In this way any machine can be added to an HTCondorpool. The dynamic deployment tool set also provides tools to remove a machine from the pool,without leaving residual effects on the machine such as leftover installations, log files, and workingdirectories.Installation and start up is provided by condor_cold_start. The condor_cold_start program determinesthe operating system and architecture of the target machine, and transfers the correct installationpackage from an ftp, http, or grid ftp site. After transfer, it installs HTCondor and createsa local working directory for HTCondor to run in. As a last step, condor_cold_start begins runningHTCondor in a manner which allows for later easy and reliable shut down.The program that reliably shuts down and uninstalls a previously dynamically installed HTCondorinstance is condor_cold_stop. condor_cold_stop begins by safely and reliably shutting off therunning HTCondor installation. It ensures that HTCondor has completely shut down before continuing,and optionally ensures that there are no queued jobs at the site. Next, condor_cold_stopremoves and optionally archives the HTCondor working directories, including the log directory.These archives can be stored to a mounted file system or to a grid ftp site. As a last step, condor_cold_stop uninstalls the HTCondor executables and libraries. The end result is that the machineresources are left unchanged after a dynamic deployment of HTCondor leaves.Configuration and UsageDynamic deployment is designed for the expert HTCondor user and administrator. Tool designchoices were made for functionality, not ease-of-use.Like every installation of HTCondor, a dynamically deployed installation relies on a configuration.To add a target machine to a previously created HTCondor pool, the global configuration filefor that pool is a good starting point. Modifications to that configuration can be made in a separate,local configuration file used in the dynamic deployment. The global configuration file must be placedon an ftp, http, grid ftp, or file server accessible by condor_cold_start. The local configuration fileis to be on a file system accessible by the target machine. There are some specific configurationvariables that may be set for dynamic deployment. A list of executables and directories which mustbe present for HTCondor to start on the target machine may be set with the configuration variablesDEPLOYMENT_REQUIRED_EXECS and DEPLOYMENT_REQUIRED_DIRS . If defined and thecomma-separated list of executables or directories are not present, then condor_cold_start exits witherror. Note this does not affect what is installed, only whether start up is successful.A list of executables and directories which are recommended to be present for HTCondorto start on the target machine may be set with the configuration variablesDEPLOYMENT_RECOMMENDED_EXECS and DEPLOYMENT_RECOMMENDED_DIRS . If definedand the comma-separated lists of executables or directories are not present, then condor_cold_startprints a warning message and continues. Here is a portion of the configuration relevant to a dynamicdeployment of a HTCondor submit node:HTCondor Version 8.0.2 Manual3.2. Installation 163DEPLOYMENT_REQUIRED_EXECS = MASTER, SCHEDD, PREEN, STARTER, \STARTER_STANDARD, SHADOW, \SHADOW_STANDARD, GRIDMANAGER, GAHP, CONDOR_GAHPDEPLOYMENT_REQUIRED_DIRS = SPOOL, LOG, EXECUTEDEPLOYMENT_RECOMMENDED_EXECS = CREDDDEPLOYMENT_RECOMMENDED_DIRS = LIB, LIBEXECAdditionally, the user must specify which HTCondor services will be started. This is donethrough the DAEMON_LIST configuration variable. Another excerpt from a dynamic submit nodedeployment configuration:DAEMON_LIST = MASTER, SCHEDDFinally, the location of the dynamically installed HTCondor executables is tricky to set, since thelocation is unknown before installation. Therefore, the variable DEPLOYMENT_RELEASE_DIR isdefined in the environment. It corresponds to the location of the dynamic HTCondor installation. If,as is often the case, the configuration file specifies the location of HTCondor executables in relationto the RELEASE_DIR variable, the configuration can be made dynamically deployable by settingRELEASE_DIR to DEPLOYMENT_RELEASE_DIR asRELEASE_DIR = $(DEPLOYMENT_RELEASE_DIR)In addition to setting up the configuration, the user must also determine where the installationpackage will reside. The installation package can be in either tar or gzipped tar form, and mayreside on a ftp, http, grid ftp, or file server. Create this installation package by taring up the binariesand libraries needed, and place them on the appropriate server. The binaries can be tared in a flatstructure or within bin and sbin. Here is a list of files to give an example structure for a dynamicdeployment of the condor_schedd daemon.% tar tfz latest-i686-Linux-2.4.21-37.ELsmp.tar.gzbin/bin/condor_config_valbin/condor_qsbin/sbin/condor_preensbin/condor_shadow.stdsbin/condor_starter.stdsbin/condor_scheddsbin/condor_mastersbin/condor_gridmanagersbin/gahp_serversbin/condor_startersbin/condor_shadowsbin/condor_c-gahpsbin/condor_offHTCondor Version 8.0.2 Manual3.3. Configuration 1643.3 ConfigurationThis section describes how to configure all parts of the HTCondor system. General informationabout the configuration files and their syntax is followed by a description of settings that affect allHTCondor daemons and tools. The settings that control the policy under which HTCondor willstart, suspend, resume, vacate or kill jobs are described in section 3.5 on Policy Configuration forthe condor_startd.3.3.1 Introduction to Configuration FilesThe HTCondor configuration files are used to customize how HTCondor operates at a given site.The basic configuration as shipped with HTCondor works well for most sites.Several sample configuration files have been published, in addition to the basicconfiguration shipped with HTCondor. Example configurations are available underthe heading of Helpful guides to new users running HTCondor at https://htcondorwiki.cs.wisc.edu/index.cgi/wiki?p=GettingStarted.Each HTCondor program will, as part of its initialization process, configure itself by callinga library routine which parses the various configuration files that might be used, including poolwide,platform-specific, and machine-specific configuration files. Environment variables may alsocontribute to the configuration.The result of configuration is a list of key/value pairs. Each key is a configuration variablename, and each value is a string literal that may utilize macro substitution (as defined below). Someconfiguration variables are evaluated by HTCondor as ClassAd expressions; some are not. Consultthe documentation for each specific case. Unless otherwise noted, configuration values that areexpected to be numeric or boolean constants may be any valid ClassAd expression of operators onconstants. Example:MINUTE = 60HOUR = (60 * $(MINUTE))SHUTDOWN_GRACEFUL_TIMEOUT = ($(HOUR)*24)Ordered Evaluation to Set the ConfigurationMultiple files, as well as a programs environment variables determine the configuration. The orderin which attributes are defined is important, as later definitions override existing definitions. Theorder in which the (multiple) configuration files are parsed is designed to ensure the security of thesystem. Attributes which must be set a specific way must appear in the last file to be parsed. Thisprevents both the naive and the malicious HTCondor user from subverting the system through itsconfiguration. The order in which items are parsed is1. global configuration fileHTCondor Version 8.0.2 Manual3.3. Configuration 1652. local configuration file3. specific environment variables prefixed with _CONDOR_The locations for these files are as given in section 3.2.2 on page 144.Some HTCondor tools utilize environment variables to set their configuration. These tools searchfor specifically-named environment variables. The variables are prefixed by the string _CONDOR_or _condor_. The tools strip off the prefix, and utilize what remains as configuration. As the use ofenvironment variables is the last within the ordered evaluation, the environment variable definitionis used. The security of the system is not compromised, as only specific variables are considered fordefinition in this manner, not any environment variables with the _CONDOR_ prefix.Configuration File MacrosMacro definitions are of the form: = The macro name given on the left hand side of the definition is a case sensitive identifier. Theremust be white space between the macro name, the equals sign (=), and the macro definition. Themacro definition is a string literal that may utilize macro substitution.Macro invocations are of the form:$(macro_name)Macro definitions may contain references to other macros, even ones that are not yet defined, aslong as they are eventually defined in the configuration files. All macro expansion is done after allconfiguration files have been parsed, with the exception of macros that reference themselves.A = xxxC = $(A)is a legal set of macro definitions, and the resulting value of C is xxx. Note that C is actually boundto $(A), not its value.As a further example,A = xxxC = $(A)A = yyyHTCondor Version 8.0.2 Manual3.3. Configuration 166is also a legal set of macro definitions, and the resulting value of C is yyy.A macro may be incrementally defined by invoking itself in its definition. For example,A = xxxB = $(A)A = $(A)yyyA = $(A)zzzis a legal set of macro definitions, and the resulting value of A is xxxyyyzzz. Note that invocationsof a macro in its own definition are immediately expanded. $(A) is immediately expanded in line3 of the example. If it were not, then the definition would be impossible to evaluate.Recursively defined macros such asA = $(B)B = $(A)are not allowed. They create definitions that HTCondor refuses to parse.All entries in a configuration file must have an operator, which will be an equals sign (=). Identifiersare alphanumerics combined with the underscore character, optionally with a subsystem nameand a period as a prefix. As a special case, a line without an operator that begins with a left squarebracket will be ignored. The following two-line example treats the first line as a comment, andcorrectly handles the second line.[HTCondor Settings]my_classad = [ foo=bar ]To simplify pool administration, any configuration variable name may be prefixed by a subsystem(see the $(SUBSYSTEM) macro in section 3.3.1 for the list of subsystems) and the period (.)character. For configuration variables defined this way, the value is applied to the specific subsystem.For example, the ports that HTCondor may use can be restricted to a range using the HIGHPORTand LOWPORT configuration variables.MASTER.LOWPORT = 20000MASTER.HIGHPORT = 20100Note that all configuration variablesmay utilize this syntax, but nonsense configuration variablesmay result. For example, it makes no sense to defineNEGOTIATOR.MASTER_UPDATE_INTERVAL = 60since the condor_negotiator daemon does not use the MASTER_UPDATE_INTERVAL variable.It makes little sense to do so, but HTCondor will configure correctly with a definition such asHTCondor Version 8.0.2 Manual3.3. Configuration 167MASTER.MASTER_UPDATE_INTERVAL = 60The condor_master uses this configuration variable, and the prefix of MASTER. causes this configurationto be specific to the condor_master daemon.This syntax has been further expanded to allow for the specification of a local name on thecommand line using the command line option-local-name This allows multiple instances of a daemon to be run by the same condor_master daemon, eachinstance with its own local configuration variable.The ordering used to look up a variable, called :1. ..2. .3. .4. If this local name is not specified on the command line, numbers 1 and 2 are skipped. As soonas the first match is found, the search is completed, and the corresponding value is used.This example configures a condor_master to run 2 condor_schedd daemons. The condor_masterdaemon needs the configuration:XYZZY = $(SCHEDD)XYZZY_ARGS = -local-name xyzzyDAEMON_LIST = $(DAEMON_LIST) XYZZYDC_DAEMON_LIST = + XYZZYXYZZY_LOG = $(LOG)/SchedLog.xyzzyUsing this example configuration, the condor_master starts up a second condor_schedd daemon,where this second condor_schedd daemon is passed -local-name xyzzy on the command line.Continuing the example, configure the condor_schedd daemon named xyzzy. This condor_schedd daemon will share all configuration variable definitions with the other condor_schedddaemon, except for those specified separately.SCHEDD.XYZZY.SCHEDD_NAME = XYZZYSCHEDD.XYZZY.SCHEDD_LOG = $(XYZZY_LOG)SCHEDD.XYZZY.SPOOL = $(SPOOL).XYZZYHTCondor Version 8.0.2 Manual3.3. Configuration 168Note that the example SCHEDD_NAME and SPOOL are specific to the condor_schedd daemon,as opposed to a different daemon such as the condor_startd. Other HTCondor daemons using thisfeature will have different requirements for which parameters need to be specified individually.This example works for the condor_schedd, and more local configuration can, and likely would bespecified.Also note that each daemons log file must be specified individually, and in two places: onespecification is for use by the condor_master, and the other is for use by the daemon itself. Inthe example, the XYZZY condor_schedd configuration variable SCHEDD.XYZZY.SCHEDD_LOGdefinition references the condor_master daemons XYZZY_LOG.Comments and Line ContinuationsAn HTCondor configuration file may contain comments and line continuations. A comment isany line beginning with a pound character (#). A continuation is any entry that continues acrossmultiples lines. Line continuation is accomplished by placing the backslash character (\) at the endof any line to be continued onto another. Valid examples of line continuation areSTART = (KeyboardIdle > 15 * $(MINUTE)) && \((LoadAvg - CondorLoadAvg)