Standards for ETL UNIX Shell Scripts for use with PowerMart 6.2 7/28/03 K.

Gillenwater (Updated 10/6/04)*

Background: The upgrade of the Development and Production PowerMart repositories and servers necessitates changes be made to ETL UNIX shell scripts used with a job scheduling tool. These changes, in large part, due to the location of PowerMart installation directories and PowerMart command line syntax changes provide an opportunity to not only change existing scripts, but also provide new standards for script development. The following provides information as to what current scripts contain, and the information that needs to be changed/included in the updated/new scripts. These standards need be followed to ensure that the jobs work in the intended fashion. Current Scripting (prior to the upgrade of PowerMart): The following example is a UNIX shell script used prior to the upgrade of the PowerMart server to version 6.2:
export SHLIB_PATH=$ORACLE_HOME/lib:/usr/local/pmserver/informatica/pm/ export MACHINE=`hostname` cd /usr/local/pmserver/informatica/pm/ pmcmd start <username> <password> $MACHINE:4001 bs_siqrefqa_refresh 0 1

There are a couple of items that should be noted in the script. One, the first three lines of the script set up variables and change the directory to point to the PowerMart installation directory where the pmcmd executable resides to start the scheduled job and run pmserver. Secondly, the pmcmd command line lists the username and shows the password. You will note in the following section outlining the new standards that there are several changes to remove much of the hard coding in the scripts for ease of maintenance and to remove the passwords from the script itself. New Scripting Standards (for use with PowerMart version 6.2): New scripting standards include the use of a UNIX shell script which the scheduling tool uses to start the PowerMart job, and a separate file which contains the username and password for the user called in the script. The following is a template that should be used to create scripts for new scheduled jobs, as well as updating current jobs. Please note that once the upgrade is complete, existing scripts will no longer function. All scripts will need to be updated prior to the upgrade of the Development and Production instances of PowerMart. Following is the script and explanation. This script has been provided for example and is named etl_unix_shell_script.sh.
# # # # # # # # # Program: Author: Date: Purpose: etl_unix_shell_script.sh Kevin Gillenwater 7/24/2003 Sample UNIX shell script to load environment variables needed to run PowerMart jobs, pass username and password variables and start the job using the pmcmd command line. $1 = Project Id Parameter (ie. ud, hr, dss, siq, etc.)

# # # # # #

Example usage: etl_unix_shell_script.sh dss NOTE: Enter the Project ID parameter that is designated in the directory structure for your team (ie. dss would be used for the DWDAS team as the directory is /usr/local/autopca/dss/)

#----------------------------------------------------------------# Call the script to set up the Informatica Environment Variables: #----------------------------------------------------------------. /usr/local/bin/set_pm_var.sh #----------------------------------------------------------------# Read ETL configuration parameters from a separate file: #----------------------------------------------------------------ETL_CONFIG_FILE=$JOBBASE/$1/remote_accts/test_etl.config ETL_USER=`grep ETL_USER $ETL_CONFIG_FILE | awk -F: '{print $2}'` ETL_PWD=`grep ETL_PWD $ETL_CONFIG_FILE | awk -F: '{print $2}'` #----------------------------------------------------------------# Start the job #----------------------------------------------------------------$PM_HOME/pmcmd startworkflow -u $ETL_USER -p $ETL_PWD -s $MACHINE:4001 -f DWDAS_ LOAD_dssqa wait s_m_CENTER_INSTITUTE #----------------------------------------------------------------# Clean up script to remove files older than 30 days #----------------------------------------------------------------/usr/bin/find /usr/local/autopca/dss/tmp -mtime +30 -exec rm {} \;

Notes Regarding the Script/Standards: 1. The beginning of each script should call and execute the script set_pm_var.sh to set up the PowerMart variables used in the session (. /usr/local/bin/set_pm_var.sh). The set_pm_var.sh located on each machine (Leisure for Production and Glance for Development) will provide for ease of maintenance if changes need to be made to PowerMart variables, and will provide for one source for the scripts on a machine. The following is the code in the script. You will not need to do anything with this script, it is for informational purposes only:
# # # # # Program: set_pm_var.sh Author: Kevin Gillenwater Date: 7/3/2003 Purpose: UNIX script which sets the variables for running PowerMart 6.2 when called from a shell script (ie. script run by Autosys).

#------------------------------------------#Set up the Informatica Variables #------------------------------------------export MACHINE=`hostname` export SHLIB_PATH=/usr/local/pmserver/informatica/pm/infoserver export PM_HOME=/usr/local/pmserver/informatica/pm/infoserver #--------------------------------------------------------------------# Set the environment variables needed for scheduling jobs.

# The value of JOBBASE differs based on the account. For AUTOSYS # and AUTODBA, the variable should evaluate to /usr/local/autopca. # For all other accounts, it should evaluate to their $HOME variable. #--------------------------------------------------------------------case $HOME in /home/autopca/autopca) /home/autopca/autodba) *) esac export JOBBASE JOBBASE=/usr/local/autopca JOBBASE=/usr/local/autopca JOBBASE=$HOME ;; ;; ;;

2. The second section of etl_unix_shell_script.sh sets up ETL parameters for username and password usage by PowerMart when running the workflow. The username and password are no longer stored as part of the main shell script. In the script example, the filename test_etl_config contains the username and password to be used when running the workflow. Following is the contents of test_etl_config:
ETL_USER:etlguy ETL_PWD:ou812

This file (your password file) must be located in your team directory under the remote_accts folder (i.e. /usr/local/autopca/dss/remote_accts/). The folder permissions on the Production machine will only permit PCAs access to this folder. The permissions should be 6-4-0 (rw,r,-) on this file. 3. The third section of etl_unix_shell_script.sh contains the command to run the workflow in PowerMart. Please follow the script exactly. The only thing that will need to be changed in this section is the folder name and the workflow within the folder that is being executed (i.e. ... DWDAS_LOAD_dssqa s_m_CENTER_INSTITUTE). 4. The final section of etl_unix_shell_script.sh contains code to clean up the scheduling tools log files that are older than 30 days according to standards. Simply change the directory to point to your teams specific folder (i.e. /usr/local/autopca/dss/tmp ).

*8/27/03 Added -wait parameter to the pmcmd command line to start the scheduled job. *10/6/04 - Updated to generalize the scheduling tool used to run the shell scripts as UC4 has been chosen to replace Autosys as the scheduling tool used to run PowerMart workflows.