unix shell script standards for powermart 622

8/6/2019 UNIX Shell Script Standards for PowerMart 622

1/3

Standards for ETL UNIX Shell Scripts for use with PowerMart 6.27/28/03 K. Gillenwater (Updated 10/6/04)*

Background:

The upgrade of the Development and Production PowerMart repositories and servers necessitates changes be made to ETL UNIX shell scripts used with a job scheduling tool. These changes, in large part, due tothe location of PowerMart installation directories and PowerMart command line syntax changes providean opportunity to not only change existing scripts, but also provide new standards for script development.The following provides information as to what current scripts contain, and the information that needs to

be changed/included in the updated/new scripts. These standards need be followed to ensure that the jobswork in the intended fashion.

Current Scripting (prior to the upgrade of PowerMart):

The following example is a UNIX shell script used prior to the upgrade of the PowerMart server to

version 6.2:export SHLIB_PATH=$ORACLE_HOME/lib:/usr/local/pmserver/informatica/pm/export MACHINE=`hostname`

cd /usr/local/pmserver/informatica/pm/

pmcmd start $MACHINE:4001 bs_siqrefqa_refresh 0 1

There are a couple of items that should be noted in the script. One, the first three lines of the script set upvariables and change the directory to point to the PowerMart installation directory where the pmcmdexecutable resides to start the scheduled job and run pmserver. Secondly, the pmcmd command line liststhe username and shows the password. You will note in the following section outlining the new standardsthat there are several changes to remove much of the hard coding in the scripts for ease of maintenanceand to remove the passwords from the script itself.

New Scripting Standards (for use with PowerMart version 6.2):

New scripting standards include the use of a UNIX shell script which the scheduling tool uses to start thePowerMart job, and a separate file which contains the username and password for the user called in thescript. The following is a template that should be used to create scripts for new scheduled jobs, as well asupdating current jobs. Please note that once the upgrade is complete, existing scripts will no longer function. All scripts will need to be updated prior to the upgrade of the Development and Productioninstances of PowerMart. Following is the script and explanation. This script has been provided for example and is named etl_unix_shell_script.sh .

# Program: etl_unix_shell_script.sh# Author: Kevin Gillenwater# Date: 7/24/2003# Purpose: Sample UNIX shell script to load environment variables# needed to run PowerMart jobs, pass username and password variables# and start the job using the pmcmd command line.## $1 = Project Id Parameter (ie. ud, hr, dss, siq, etc.)#


2/3

# Example usage: etl_unix_shell_script.sh dss## NOTE: Enter the Project ID parameter that is designated in the# directory structure for your team# (ie. dss would be used for the DWDAS team as the# directory is /usr/local/autopca/dss/)

#-----------------------------------------------------------------# Call the script to set up the Informatica Environment Variables:#-----------------------------------------------------------------

. /usr/local/bin/set_pm_var.sh

#-----------------------------------------------------------------# Read ETL configuration parameters from a separate file:#-----------------------------------------------------------------

ETL_CONFIG_FILE=$JOBBASE/$1/remote_accts/test_etl.configETL_USER=`grep ETL_USER $ETL_CONFIG_FILE | awk -F: '{print $2}'`ETL_PWD=`grep ETL_PWD $ETL_CONFIG_FILE | awk -F: '{print $2}'`

#-----------------------------------------------------------------# Start the job#-----------------------------------------------------------------

$PM_HOME/pmcmd startworkflow -u $ETL_USER -p $ETL_PWD -s $MACHINE:4001 -f DWDAS_LOAD_dssqa wait s_m_CENTER_INSTITUTE

#-----------------------------------------------------------------# Clean up script to remove files older than 30 days#-----------------------------------------------------------------

/usr/bin/find /usr/local/autopca/dss/tmp -mtime +30 -exec rm {} \;

Notes Regarding the Script/Standards:

1. The beginning of each script should call and execute the script set_pm_var.sh to set up the PowerMartvariables used in the session (. /usr/local/bin/set_pm_var.sh). The set_pm_var.sh located on each machine(Leisure for Production and Glance for Development) will provide for ease of maintenance if changesneed to be made to PowerMart variables, and will provide for one source for the scripts on a machine. Thefollowing is the code in the script. You will not need to do anything with this script, it is for informational

purposes only:

# Program: set_pm_var.sh# Author: Kevin Gillenwater# Date: 7/3/2003# Purpose: UNIX script which sets the variables for running PowerMart 6.2# when called from a shell script (ie. script run by Autosys).

#-------------------------------------------#Set up the Informatica Variables#-------------------------------------------

export MACHINE=`hostname`export SHLIB_PATH=/usr/local/pmserver/informatica/pm/infoserverexport PM_HOME=/usr/local/pmserver/informatica/pm/infoserver

#---------------------------------------------------------------------# Set the environment variables needed for scheduling jobs.


3/3

# The value of JOBBASE differs based on the account. For AUTOSYS# and AUTODBA, the variable should evaluate to /usr/local/autopca.# For all other accounts, it should evaluate to their $HOME variable.#---------------------------------------------------------------------

case $HOME in

/home/autopca/autopca) JOBBASE=/usr/local/autopca ;;/home/autopca/autodba) JOBBASE=/usr/local/autopca ;;*) JOBBASE=$HOME ;;

esac

export JOBBASE

2. The second section of etl_unix_shell_script.sh sets up ETL parameters for username and passwordusage by PowerMart when running the workflow. The username and password are no longer stored as

part of the main shell script. In the script example, the filename test_etl_config contains the username and password to be used when running the workflow. Following is the contents of test_etl_config :

ETL_USER:etlguyETL_PWD:ou812

This file (your password file) must be located in your team directory under the remote_accts folder (i.e./usr/local/autopca/dss/remote_accts/). The folder permissions on the Production machine will only permitPCAs access to this folder. The permissions should be 6-4-0 (rw,r,-) on this file.

3. The third section of etl_unix_shell_script.sh contains the command to run the workflow inPowerMart. Please follow the script exactly. The only thing that will need to be changed in this section isthe folder name and the workflow within the folder that is being executed(i.e. ... DWDAS_LOAD_dssqa s_m_CENTER_INSTITUTE ).

4. The final section of etl_unix_shell_script.sh contains code to clean up the scheduling tools log filesthat are older than 30 days according to standards. Simply change the directory to point to your teamsspecific folder (i.e. /usr/local/autopca/dss/tmp ).

*8/27/03 Added -wait parameter to the pmcmd command line to start the scheduled job.*10/6/04 - Updated to generalize the scheduling tool used to run the shell scripts as UC4 has been chosento replace Autosys as the scheduling tool used to run PowerMart workflows.

unix shell script standards for powermart 622

Documents