Class10 - Bash Shell Scripting PDF

Bash Survival Guide for HPC
M. D. Jones, Ph.D.
Center for Computational Research University at Buffalo State University of New York
High Performance Computing I, 2012
M. D. Jones, Ph.D. (CCR/UB)
HPC-I Fall 2012
1 / 52
Introduction
A Little History
History of bash
The concept of the shell environment has been around for a long time (long = lifetime of UNIX operating system): bash is a twist on one of the earliest UNIX shells (the so-called Bourne shell, sh on most systems). bash is a superset of the (limited) Bourne shell syntax There is quite an extended family of shells, a good link for which can be found here:
http://en.wikipedia.org/wiki/Comparison_of_command_shells
HPC-I Fall 2012
3 / 52
Introduction
Feature Subset
bash Features
Subset of important features of bash: Basically a command line interface (CLI) to the operating system Command line completion (via TAB key) Command line editing, history Output redirection Control constructs ... Basically all of the shells give you the ability to enter text-based commands/queries much faster than using a GUI (if your particular OS even supports a GUI).
HPC-I Fall 2012
4 / 52
Introduction
Startup Scripts
bash Invocation/Startup Scripts

bash will read from various startup scripts depending on how it was invoked: Interactive login shell: uses /etc/profile, then ~/.bash_profile, ~/.bash_login, ~/.profile in that order (if they are present at all). Interactive non-login shell: ~/.bashrc (suppressed by --norc option, or --rcfile le alternative) Non-interactive: uses $BASH_ENV (subject to substitution, if result is a le, executes the le) as sh: attempts to duplicate Bourne shell behavior, reads and executes /etc/profile and ~/.profile, in order via rshd: will read and execute ~/.bashrc (if it is able to determine that is was launched via rsh)
M. D. Jones, Ph.D. (CCR/UB) Bash Survival Guide for HPC HPC-I Fall 2012 5 / 52
Introduction
Navigation
CLI Navigation
bash supports a rich set of line editing features making it easier for you to move the cursor around the command line, recall past commands and edit them, etc. The default syntax follows that of the emacs text editor (you can customize the syntax, of course, see man bash). Very briey, the arrow keys can typically be used to navigate the current command line (left and right arrow keys, up and down to scroll though past commands), plus some common key bindings (there are many more than this small sample):
C-a go to the beginning of the line C-e go to the end of the line C-f,C-b go forward,back one character M-f,M-b go forward,back one word
where C- denotes the Control key on the keyboard, and M- the META key - not many keyboards have META keys these days, in which case ESC is used instead.
HPC-I Fall 2012
6 / 52
Introduction
Navigation
CLI History
Unsurprisingly, the shell records a history of what commands you execute. The history command will list the commands in your shells history with an identication number that you can use as shorthand for recalling them:
[ u2 : ~ ] $ ! ! # r e p e a t v e r y l a s t command l s l [ u2 : ~ ] $ h i s t o r y # s p i t s o u t l a s t HISTSIZE commands [ u2 : ~ ] $ !1574 # p i c k one by number , p r e f a c e w i t h ! who am i jonesm pts /23 20120525 12:53 ( cash . c c r . b u f f a l o . edu ) [ u2 : ~ ] $ ! 1 5 6 2 : s/x/a / # p i c k one , use a regex t o modify on t h e f l y pbsnodes a | l e s s
HPC-I Fall 2012
7 / 52
Introduction
Environment Variables
Environment variables are heavily used by all shells to set the user environment and control its behavior, and are also frequently used by applications. For now, here are just a few of the simplest of the defaults: $HOME , your home directory location. $PATH , search path for commands $LD_LIBRARY_PATH , search path for dynamic libraries $MANPATH , search path for man pages $PS1 , command prompt syntax Note that the modules environment package used in CCR is designed to allow you to more easily manipulate the environment (especially $PATH, $LD_LIBRARY_PATH, and $MANPATH) for various software packages.
Introduction
You can set environment variables very simply by assignment or using the export command, the difference being whether the value is passed on to sub-shells (i.e. if you run another script or executable) you can also run a program with specic settings using the env command:
# these v a r i a b l e s are s e t o n l y i n t h e c u r r e n t s h e l l NP= ` cat $PBS_NODEFILE | wc l ` # counts l i n e s i n $PBS_NODEFILE NODES= ` cat $PBS_NODEFILE | uniq ` # counts unique l i n e s i n $PBS_NODEFILE # t h i s applies to a l l children of t h i s shell # ( I n t e l MKL core a f f i n i t y s e t t i n g ) export KMP_AFFINITY=nowarnings , compact # t h i s runs a s h e l l s c r i p t u s i n g a l o c a l s e t t i n g f o r $HOME # (MATLAB compiled b i n a r i e s l i k e t o abuse $HOME d i r e c t o r i e s . . . ) env HOME= / tmp . / r u n _ e x t r a c t . sh
You can add common settings to your $HOME/.bashrc le if you want them to apply to all of your logins/shells. Using export without any arguments will print the current settings for your shell (as will the env command).
HPC-I Fall 2012
9 / 52
Basic Syntax
Format
The First Line
Shell scripts can be no more than an ordered list of commands to run. By convention, the rst line declares the shell:
# ! / b i n / bash # # pound symbol g e n e r a l l y s t a r t s a comment , except f o r t h a t f i r s t echo ' H e l l o , w o r l d ! ' line
HPC-I Fall 2012
11 / 52
Basic Syntax
Format
Variables
Variables in bash are easily declared:
# some v a r i a b l e d e c l a r a t i o n s : s t r 1 = " Goodbye , w o r l d ! " # note quotes t o p r o t e c t t h e space s t r 2 = ` date ` # b a c k t i c k s t o execute commands , # capture the output echo n '$USER = ' # s i n g l e quotes do n o t exand $USER # and n does n o t break l i n e echo "$USER" # double quotes w i l l expand $USER # p r e c e d i n g two l i n e s p r i n t o u t $USER =johndoe f o r u s e r i d johndoe # t h e r e are a bunch o f p r e d e f i n e d v a r i a b l e s , $USER i s one echo " $ { s t r 1 } Aargh ! " # braces are handy t o p r o t e c t v a r i a b l e s str3=" " i f [ n " $ s t r 3 " ] ; then # n i n d i c a t e s nonempty argument echo ' $ s t r 3 i s n o t empty ! ' fi
That last bit is a good example of why variables are often enclosed in quotes (they do not have to be), in case they evaluate as the empty string.
HPC-I Fall 2012
12 / 52
Basic Syntax
Command Substitution
Command Substitution
Command substitution is an invaluable way to access the output of a command from the environment. You can do this either with backticks, command, or subshells, $(commands):
f i l e s ="$( ls ) " h t m l _ f i l e s = ` l s / v a r /www/ html ` i n d e x = ` expr 4 * 2 + 1 ` # l i s t i n g o f cwd # l i s t i n g o f / v a r /www/ h t m l # use expr t o e v a l u a t e math
Note that $() is very easy to nest, which gives it an edge over backticks. Newlines are not allowed, so they are stripped out in both methods.
HPC-I Fall 2012
13 / 52
Basic Syntax
Arithmetic Expansion
Arithmetic
bash has no notion of oating-point math, although you can resort to external tools like bc to work around that lack. Integer arithmetic is done using backticks, double parentheses, or let:
z = ` expr $z + 1 ` z=$ ( ( $z + 1 ) ) z=$ ( ( z + 1 ) ) ( ( z += 1 ) ) l e t z=z+1 l e t " z += 1 " # expr command does t h e work # # # # parameter der e f e r e n c i n g o p t i o n a l i n s i d e double parentheses shortcut l e t command quotes l e t you use spaces
HPC-I Fall 2012
14 / 52
Basic Syntax
I/O Redirection
I/O Redirection
Very handy topic - how to redirect input/output in bash. First, though, there are three default les: stdin (keyboard), descriptor 0 stdout (screen), descriptor 1 stderr (error messages to the screen), descriptor 2 Redirection just means capturing output from a le, command, or script (or portion of a script) and sending it elsewhere as input. Note that descriptors 3-9 are also available for use.
HPC-I Fall 2012
15 / 52
Basic Syntax
I/O Redirection
Generally you can redirect input with <, and output with | (pipe) or > (le), but there are quite a few more variations best illustrated by example:
echo ' blah ' > f i l e n a m e echo ' blah ' >> f i l e n a m e . / somecode 1> f i l e n a m e . / somecode 2> f i l e n a m e . / somecode &> f i l e n a m e . / somecode > f i l e n a m e 2>&1 . / othercode < f i l e n a m e . / code < inname &> outname command1 | command2 | command3 # # # # # # # # # # cat * . t x t | s o r t | u n i q > r e s u l t # overwrites filename appends f i l e n a m e f i l e n a m e c a p t u r e s t d o u t from somecode f i l e n a m e c a p t u r e s s t d e r r from somecode f i l e n a m e c a p t u r e s both s t d o u t and s t d e r r same as above , o l d ( Bourne ) s t y l e othercode g e t s i n p u t from f i l e n a m e c o mb i na t io n s are ok p i p e s are s i m i l a r t o > b u t more g e n e r a l f o r c h a i n i n g commands t o g e t h e r see?
HPC-I Fall 2012
16 / 52
Control Constructs
Conditionals
if--fi
bash has a conditional construct, of course:
i f conditionA then statement0 statement1 ... e l i f conditionB then morestatements1 morestatements2 ... else yetmore0 yetmore1 ... fi
We will go into the sytnax for the conditions next, but note that the conditional branches are often written more concisely using the ";" separator:
i f c o n d i t i o n A ; then listofstatements ... fi
HPC-I Fall 2012
18 / 52
Control Constructs
Conditionals
Is There in Truth No Beauty?
The test command (and its shorthand version) gets used quite often in conditionals (and in general):
# f i r s t form t e s t operand1 o p e r a t o r operand2 # example : i f t e s t n "$PBS_ENVIRONMENT" ; then echo " Hey , I am i n a PBS environment ! " fi # second form [ operand1 o p e r a t o r operand2 ] # example : i f [ n "$PBS_ENVIRONMENT" ] ; then echo " Hey , I am s t i l l i n a PBS environment ! " fi # I cheated a b i t , n i s a unary o p e r a t o r . . .
HPC-I Fall 2012
19 / 52
Control Constructs
Conditionals
Operators
I am not going to show the full set of test operators (see man bash), but here is a quick subset:
Operator -n -z -f -d == != <,> -eq,-ne -lt,-gt -le,-ge || && Operands 1 1 1 1 2 2 2 2 2 2 2 2 nonzero length zero length le given by operand exists directory given by operand exists string equality string lexical integer (in)equality integer comparisons integer comparisons logical OR logical AND
Note the absence of oating-point operations (serious shortcoming).
HPC-I Fall 2012
20 / 52
Control Constructs
Loop Constructs
for Loops
First of the main loop types in bash, for loops repeat over a simple index of space separated items:
f o r name i n " $LIST " do echo " working on $name " bunchofstatements1 done
So the loop counter, name is set to each item in the list as the loop is executed. Note that the familiar form from C for integer indices is also valid:
f o r ( ( expr1 ; expr2 ; expr3 ) ) ; do l i s t o f s t a t e m e n t s ; done # f o r example : f o r ( ( i =1; i <10; i + + ) ) do echo " i = $ i " done
Note that bash is pretty slow (most interpreted languages are slow), so the C-like loop syntax is not encouraged.
Control Constructs
Loop Constructs
Globbing
You can make handy use of substitution by globbing, or using * to match all lenames.
# s i m p l e g l o b b i n g example f o r fname i n * . c # grabs a l l . c f i l e s i n c u r r e n t d i r e c t o r y do grep ' double ' $fname # p u l l a l l o u t a l l t h e doubles done
HPC-I Fall 2012
22 / 52
Control Constructs
Loop Constructs
while Loops
While loops are just about what you would expect:

while l i s t ; do l i s t o f s t a t e m e n t s ; done # s i m p l e example while [ " $ c o u n t e r " l t " $UPPER_LIMIT " ] ; do echo " c o u n t e r = $ c o u n t e r " c o u n t e r = ` expr $ c o u n t e r + 1 ` # c o u n t e r =$ ( ( $ c o u n t e r + 1 ) ) should a l s o work # l e t " c o u n t e r += 1" should a l s o work done
There is also an until variant of while that has the same syntax (just negates the test).
HPC-I Fall 2012
23 / 52
Miscellaneous Advanced Features
Conditional Execution
Conditional Execution
A very typical bash-ism is conditional execution, namely use the return code from a command (you know that all commands have return codes to indicate success or failure, right?):
command1 && command2 # command2 i f f command1 r e t u r n s t r u e ( 0 ) command1 | | command2 # command2 i f f command1 r e t u r n s f a l s e ( nonzero ) command1 && command2 | | command3 # command2 i f f command1 r e t u r n s t r u e # command3 i f f command1 r e t u r n s f a l s e # rm f $ t h i s f i l e && echo " $ t h i s f i l e was removed . " | | echo " $ t h i s f i l e n o t removed . " # # can check e x i t s t a t u s o f l a s t command u s i n g $? echo $?
HPC-I Fall 2012
25 / 52
Functions
Functions
bash has support for functions, general syntax:
# # general f u n c t i o n syntax # one way : function_name { # body o f f u n c t i o n i n s i d e braces statements ; # should end w i t h semicolon o r n e w l i n e } # # a n o t h e r way : function functioname ( ) # using the f u n c t i o n b u i l t i n r e q u i r e s ( ) { statement1 statement2 ... statementN return # o r an e x p l i c i t r e t u r n b u i l t i n # # handy f u n c t i o n f o r DOS/ windows s h e l l users dir () { l s F - c o l o r =auto l F - c o l o r =always "$@" | l e s s r # $@ i s $1 , $2 , } }
...
Return code is the the return code from the last statement in the function.
Functions
Function Parameters
You can pass arguments to functions (they behave like mini-scripts). They become positional parameters that you can reference as $1, $2, ...
# # f u n w i t h f u n c t i o n ( o r s c r i p t i t s e l f ) parameters # echo "My name i s $0 " # name o f t h e s c r i p t echo " I have $# arguments " # number o f arguments i s $# i f [ n " $1 " ] then echo " F i r s t argument i s $1 " # 1 s t argument fi # # An o f t e n used form f o r s c r i p t s t o check argument number # i f [ ! $# eq 2 ] ; then echo " Usage : $0 arg1 arg2 " exit 1 fi
Note that $0 remains unaffected by calling a function.
HPC-I Fall 2012
27 / 52
Functions
Bit More Function Trivia
Functions can be recursive, and can have their own local variables through the local builtin.
HPC-I Fall 2012
28 / 52
Shell Limits
Shell Limits
bash shell limits are controlled through the ulimit builtin function. Easiest way to see that is by example:
[ bono : ~ ] $ u l i m i t a core f i l e s i z e ( blocks , c ) data seg s i z e ( kbytes , d ) f i l e size ( blocks , f ) pending s i g n a l s ( i ) max l o c k e d memory ( kbytes , l ) max memory s i z e ( kbytes , m) open f i l e s (n ) pipe size (512 bytes , p ) POSIX message queues ( bytes , q ) stack size ( kbytes , s ) cpu t i m e ( seconds , t ) max user processes (u ) v i r t u a l memory ( kbytes , v ) f i l e locks (x ) unlimited unlimited unlimited 1024 32 unlimited 8192 8 819200 unlimited unlimited unlimited unlimited unlimited
The values can be numerical or one of hard, soft or unlimited. The ags -H, -S can be used to specify the hard and soft limits. Note that the stack limit is of particular interest to us in HPC, and very frequently needs to be increased from its (small) default value.
Aliases
Aliases
Aliases are basically keyboard shortcuts that you can set up and use they do not do any expansion, nor can they be recursive:
# # from my ~ / . bashrc # export EDITOR= v i # # Aliases # a l i a s rm = ' rm i ' a l i a s mv= 'mv i ' a l i a s cp = ' cp i ' a l i a s xe = ' xemacs ' a l i a s myps = ' ps u jonesm Lf ' a l i a s myjobs = ' q s t a t a | grep jonesm ' a l i a s p r o j = ' cd / p r o j e c t s / jonesm ' a l i a s wien = ' cd / p r o j e c t s / jonesm / d_wien / wien2k / u2 ' a l i a s mod= ' module ' a l i a s ml = ' module load ' a l i a s mu= ' module unload ' a l i a s mls = ' module l i s t '
HPC-I Fall 2012
30 / 52
Loading Other Scripts
Loading Other Scripts
You can load other bash scripts (or just les containing bash commands) using the source command.
# # l o a d s I n t e l c o m p i l e r environment . / o p t / i n t e l / composerxe / b i n / c o m p i l e r v a r s . sh i n t e l 6 4
Sometimes you will see the shorthand . used instead of source (same meaning).
HPC-I Fall 2012
31 / 52
Regular Expressions
Regular Expressions
Regular expressions (REs) occur very frequently in UNIX, especially in sed, awk, and grep. If you have done any signicant work in Perl, of course, you are likely also well versed in REs. REs are sets of (meta)characters used for pattern matching in text searches and string manipulation, and contain one (or more) of: characters - retain their literal meaning anchors - designate position in the text line, ^ (beginning) and $ (end) modiers - expand or narrow the range of text to search, includes , brackets, and the backslash
HPC-I Fall 2012
32 / 52
Regular Expressions
More on modiers in REs:

*, matches any number of the character string or RE preceding it ., (dot) matches any non-newline character ^, (carat) matches beginning of line, but also for negation (context dependent) $, matches end of line [,], brackets enclose a set of characters to match, including negation with ^ and ranges with \, backslash escapes a special character (e.g. $) \<,\>, escaped angle brackets match word boundaries ?, question mark matches zero or one of the preceding RE, used mostly for single characters +, matches one or more of the previous RE (like *, but does not match zero occurrences) \{,\}, escaped curly braces indicate number of previous REs to match (), parentheses enclose a group of REs |, the or RE operator used in matching alternates
HPC-I Fall 2012
33 / 52
Regular Expressions
RE Examples
Used in the context of grep just as an example:

grep " 2224 * " t e x t f i l e grep grep grep grep grep grep grep grep grep grep " 24. " t e x t f i l e " ^A" t e x t f i l e " A$ " t e x t f i l e " ^$ " t e x t f i l e " [ abc123 ] " t e x t f i l e " [ ac13] " t e x t f i l e " [ ^ ac ] " t e x t f i l e " [ Yy ] [ Ee ] [ Ss ] " t e x t f i l e " \$\$" t e x t f i l e " \ < yes \ > " t e x t f i l e # # # # # # # # # # # # matches 222 f o l l o w e d by any number o f 4s e . g . 222 , 2224 , 22244 , . . . matches 24 f o l l o w e d by any c h a r a c t e r , b u t n o t 24 matches any l i n e b e g i n n i n g w i t h A matches any l i n e ending w i t h A matches b l a n k l i n e s matches any o f t h e c h a r a c t e r a , b , c , 1 , 2 , 3 same t h i n g u s i n g ranges matches a l l c h a r a c t e r s except ac case i n s e n s i t i v e match f o r YES, yes , Yes , . . . match $$ matches o n l y d i s t i n c t word yes
REs are a rich topic in their own right ...
HPC-I Fall 2012
34 / 52
Here Documents
Here Documents
So-called here documents are ways to pass text on-the-y into programs or other code blocks. Its another form of I/O redirection, but a very handy one for running programs:
. / a . out <<ENDOFINPUT # n o t h i n g s p e c i a l about t h i s s t r i n g , j u s t needs t o be unique inputline1 # and matched below . These l i n e s appear as s t a n d a r d i n p u t inputline2 # t o t h e program a . o u t ... ENDOFINPUT # # you can p r e v e n t s u b s t i t u t i o n by q u o t i n g t h e s t r i n g # . / a . out < < 'EOI ' l i n e t h a t needs $ i n p u t t h a t c o n t a i n s $ EOI cat < < 'EOF' > n e w s c r i p t . sh # v e r y handy t o automate s c r i p t g e n e r a t i o n # ! \ b i n / bash # echo " $0 $@" ... EOF
HPC-I Fall 2012
35 / 52
Job Control
Basic Job/Process Control

You have the ability to stop and continue your processes from a fairly low level to more sophisticated tools like screen. The basics are controlled through the notion of a job, created by putting a process asynchronously into the background using the & operator, or C-z (control-z), the bg and fg commands place the process into background or foreground, jobs will list all jobs:
[ u2 : ~ ] $ emacs t e s t . c ^Z [ 1 ] + Stopped [ u2 : ~ ] $ bg [ 1 ] + emacs t e s t . c & [ u2 : ~ ] $ jobs [ 1 ] + Running [ u2 : ~ ] $ fg emacs t e s t . c [ u2 : ~ ] $ k i l l %1 # c o n t r o lz suspends job , j o b i d g i v e n i n square b r a c k e t s emacs t e s t . c # p u t i n t o background , now r u n n i n g s e p a r a t e l y from t e r m i n a l # l i s t s running jobs emacs t e s t . c & # b r i n g back t o f o r e g r o u n d i n t e r m i n a l # d i e j o b 1 , d i e note % f o r j o b i d
kill terminates jobs and processes (no % sign for process ids), if you have a stuck process (identied with the ps commands) you may need to specify a higher kill level like kill -9 pid.
Job Control
Stuff Not Covered
In the context of a bash survival guide, I skipped quite a bit of stuff: Argument processing (getopts, shift) ... List-oriented data structures Built-in functions (only talked about a small subset)
HPC-I Fall 2012
37 / 52
Additional Resources
More Resources on bash
Man page for your particular bash (denitive for your version): man bash bash Reference Manual: http://www.gnu.org/software/bash/manual Beginners guide to bash (LDP): http://tldp.org/LDP/Bash-Beginners-Guide Advanced bash Scripting (LDP): http://tldp.org/LDP/abs
HPC-I Fall 2012
38 / 52
Miscellaneous Utilities (not just for bash)
viewers
Viewers
You do not need (and often do not want) to use an editor just to look at a le (or a long listing of results, les in a directory, etc.), so Linux/UNIX provides viewing utilities that you can use: head - prints the rst N (default 10) lines of a le, e.g., head -30 filename tail - prints the last N (default 10) lines of a le, e.g., tail -30 filename cat - catenates a le, i.e. just spits it out to standard output (-n shows line numbers, -t shows non- printing characters and tabs) more - le lter, with pagination commands like vi less - alternative to more intended to be faster (does not attempt to parse the whole le) and with more freedom of movement. Cf. man less
sed
sed
sed, the stream editor, is one of the oldest of the UNIX editors (and therefore probably the least user-friendly). It has a lot of power, though, and you will still nd it heavily utilized in shell scripts. Some very simple examples:
# s e l f e x p l a n a t o r y ( note s i n g l e quotes ) # [ u2 : ~ ] $ echo ' H e l l o , w o r l d ! ' | sed s / H e l l o / Goodbye / Goodbye , w o r l d ! [ u2 : ~ ] $ echo ' H e l l o , w o r l d ! ' | sed " s / H e l l o / Goodbye , c r u e l / " Goodbye , c r u e l , w o r l d ! # note quote usage [ u2 : ~ ] $ echo ' H e l l o , w o r l d ! '
HPC-I Fall 2012
41 / 52
grep
grep
grep is a handy little utility to perform searches on les for lines containing a given string (the origin of the name dates back to UNIXs original command editor, ed, namely g/re/p. Simple examples:
[ u2 : ~ ] $ [ u2 : ~ ] $ [ u2 : ~ ] $ [ u2 : ~ ] $ w > tmp .w grep $USER tmp .w grep j o e u s e r tmp .w w | grep $USER # # # # dump o u t p u t o f w t o a temporary f i l e , c f . my c u r r e n t l o g i n s t o t h i s machine j o e u s e r ' s l o g i n s t o t h i s machine we don ' t need no s t i n k i n ' temporary f i l e s 'man w '
...
man grep for handy options (notable, -i and -n).
HPC-I Fall 2012
42 / 52
awk
awk
awk is a very powerful utility - it harbors a complex programming environment in its own right, but you are free to use only as much of it as you want or need to at a given time. The general syntax for awk looks like:
awk ' / p a t t e r n / { a c t i o n } ' f i l e
a given action will be performed on every record which matches the given /pattern/. A very handy list of the elds in the record are given as variables, $0 (entire record), $1 (rst eld), $2 (second eld) .... Lets try an example:
[ u2 : ~ ] $ q s t a t a > tmp . q [ u2 : ~ ] $ grep x d t a s tmp . q [ u2 : ~ ] $ awk ' / x d t a s / { p r i n t $0 } ' tmp . q # r e c o r d c u r r e n t s t a t e o f t h e queues # our f r i e n d grep # same t h i n g w i t h awk # more i n t e r e s t i n g example [ u2 : ~ ] $ awk ' BEGIN { cores =0} / x d t a s / { cores += $7 } END { p r i n t cores } ' tmp . q # add nodes , j o b s ?
HPC-I Fall 2012
43 / 52
awk
Exercise
Modify the preceding awk example to also count and report the number of jobs and nodes as well as cores.
HPC-I Fall 2012
44 / 52
find
find
find is quite a handy command for searching for les in complex directory layouts - see man nd for various options, the very basic syntax looks like:
f i n d [ path ] [ e x p r e s s i o n ]
where the expression involves options for just about every possible le property that you might imagine - here are a few examples:
# f i n d a l l f i l e s h e r e i n w i t h s u f f i x . nc [ u2 : / s c r a t c h / jonesm ] $ f i n d . name \ * . nc p r i n t # narrow t h e search down a b i t [ u2 : / s c r a t c h / jonesm ] $ f i n d . name 2010 \ * . nc p r i n t # l o n g l i s t each one [ u2 : / s c r a t c h / jonesm ] $ f i n d . name 2010 \ * . nc exec l s l { } \ ; # show me f i l e s i n danger o f being scrubbed from my panasas s c r a t c h d i r e c t o r y [ u2 : ~ ] $ f i n d / panasas / s c r a t c h / jonesm atime +23
Note that the xargs command is also helpful when you want to do something useful with copious amounts of output from a command like find.
And many more
Other utilities
There is a zoo of other useful commands/utilities, here is a sampling based on what I have found to be useful:
file determines a les type, typically text, data, or executable, but can also identify other common le types (e.g., it will recognize windows le types). ldd prints shared library dependencies, useful for checking to see that all dependencies are satised. cut slices les in a similar fashion to awk, but does so in a less complex way. You can specify bytes (-b), delimiters (-d), character position (-c), or elds (-f) from which to extract. paste merges lines from les (or standard input). join more exible version of paste for sources containing common elds. sort handy sorting utility (lexical, -n numeric, etc.). split splits les into pieces, by lines (-l) or by bytes (-b) or a maximum of some bytes per line (-C). tr translates or deletes characters, very useful for rapidly cleaning up les (common example, tr -d \r < infile.csv > outfile.csv, to remove windows carriage returns from a le) uniq omit (or report) repeated lines. wc newline (-l), word (-w), character (-m), or byte (-c) counts.
More CLI Resources
More CLI Resources
List of CLI related cheat sheets:

http://www.cyberciti.biz/tips/linux-unix-commands-cheat-sheets.html
The Linux documentation project:

http://tldp.org/LDP/GNU-Linux-Tools-Summary/html/GNU-Linux-Tools-Summary.html
HPC-I Fall 2012
47 / 52
Examples
Simple Tool Script
Simple Tool Script
Shell scripts are quite commonly used when deploying other tools here is an example of one that is used to wrap a Java application called Panoply1
1 2 3 4 5 6 7 8 9 10 [ u2 : ~ ] $ l s panoply / PanoplyJ c o l o r b a r s j a r s o v e r l a y s Panoply . j a r panoply . sh [ u2 : ~ ] $ cat panoply / PanoplyJ / panoply . sh # ! / b i n / sh # SCRIPT= ` r e a d l i n k f $0 ` SCRIPTDIR= ` dirname $SCRIPT ` j a v a Xmx1G Xms128m j a r $SCRIPTDIR / j a r s / Panoply . j a r "$@" README. t x t
http://www.giss.nasa.gov/tools/panoply
Bash Survival Guide for HPC HPC-I Fall 2012 49 / 52
Examples
Example PBS Script
Example PBS bash Script

A more complex example script for a scientic application on the cluster, which makes use of various utilities that we have discussed.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 #PBS S / b i n / bash #PBS q debug #PBS l 4 : IB2 : ppn=8 #PBS l w a l l t i m e =01:00:00 #PBS jonesm@ccr . b u f f a l o . edu M #PBS e m #PBS j oe #PBS N t i t a n _ 3 2 p #PBS o p b s _ s c r i p t . o u t #PBS g r o u p _ l i s t =gmfg W # # Set e x e c u t a b l e name & mpiexec path # EXE = Executable name # INITDIR = d i r h o l d i n g i n p u t & e x e c u t a b l e # SAVEDIR = d i r t o p l a c e o u t p u t f i l e s # module l o a d t i t a n / i n t e l m p i INITDIR=$PBS_O_WORKDIR SAVEDIR=$PBS_O_WORKDIR # # Get l i s t o f p ro c e s s o r s from PBS p l i s t = ` cat $PBS_NODEFILE | sed " s / \ . c c r \ . b u f f a l o \ . edu / / " ` u p l i s t = ` cat $PBS_NODEFILE | u n i q | sed " s / \ . c c r \ . b u f f a l o \ . edu / / " ` echo " unique nodes l i s t = " $ u p l i s t echo " f u l l process l i s t = " $ p l i s t
HPC-I Fall 2012
50 / 52
Examples
Example PBS Script
26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53
# Stage o u t indexed i n p u t f i l e s ( and common ones t o o ) t o each # processor cd $PBS_O_WORKDIR NN=1 f o r p i n $ p l i s t ; do l e t "NN = $NN + 1 " NUM= `echo $NN | awk ' { p r i n t f " %.4d " , $1 } ' ` echo " s t a g i n g o u t funky$NUM . i n p t o $p . . . " ssh $p " cp $INITDIR / funky$NUM . i n p $PBSTMPDIR" done # f i l e s t h a t o n l y need t o appear once on a node , no m a t t e r # how many procs / node f o r p i n $ u p l i s t ; do ssh $p " cp r $INITDIR / * . data $PBSTMPDIR / " echo " Contents o f "$PBSTMPDIR" on node " $p ssh $p " l s l $PBSTMPDIR / " done NP= ` cat $PBS_NODEFILE | wc l ` NP_UNIQ= ` cat $PBS_NODEFILE | u n i q | wc l ` # # Run code from $PBSTMPDIR make sure t h a t you copy back f i l e s # t h a t you need ($PBSTMPDIR i s erased a t c o m p l e t i o n o f t h e j o b ) # N . B . you can a l s o run t i t a n from a shared d i r e c t o r y w/ o a l l # t h e e x t r a f i l e s t a g i n g , as l o n g as t h e i / o r e q u i r e m e n t s are # r e a s o n a b l y s m a l l ( and you do n o t exceed , say , about 16p ) # cd $PBSTMPDIR mpirun np $NPROCS $EXE
HPC-I Fall 2012
51 / 52
Examples
Example PBS Script
54 55 56 57 58 59 60 61 62
# # Stage back o u t p u t f i l e s # f o r p i n $ u p l i s t ; do echo " Contents o f $PBSTMPDIR on $p : " ssh $p " l s l $PBSTMPDIR" ssh $p " cp $PBSTMPDIR / xdmf * $SAVEDIR / ; cp $PBSTMPDIR / perform * $SAVEDIR / ; \ cp $PBSTMPDIR / output_summary * $SAVEDIR / ; cp $PBSTMPDIR / p i l e h e i g h t * $SAVEDIR / " done
HPC-I Fall 2012
52 / 52

Class10 - Bash Shell Scripting PDF

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Class10 - Bash Shell Scripting PDF

Încărcat de

Drepturi de autor:

Formate disponibile

Bash Survival Guide for HPC

High Performance Computing I, 2012

M. D. Jones, Ph.D. (CCR/UB)

Bash Survival Guide for HPC

HPC-I Fall 2012

M. D. Jones, Ph.D. (CCR/UB)

Bash Survival Guide for HPC

HPC-I Fall 2012

M. D. Jones, Ph.D. (CCR/UB)

Bash Survival Guide for HPC

HPC-I Fall 2012

bash Invocation/Startup Scripts

M. D. Jones, Ph.D. (CCR/UB)

Bash Survival Guide for HPC

HPC-I Fall 2012

M. D. Jones, Ph.D. (CCR/UB)

Bash Survival Guide for HPC

HPC-I Fall 2012

M. D. Jones, Ph.D. (CCR/UB)

Bash Survival Guide for HPC

HPC-I Fall 2012

The First Line

M. D. Jones, Ph.D. (CCR/UB)

Bash Survival Guide for HPC

HPC-I Fall 2012

M. D. Jones, Ph.D. (CCR/UB)

Bash Survival Guide for HPC

HPC-I Fall 2012

M. D. Jones, Ph.D. (CCR/UB)

Bash Survival Guide for HPC

HPC-I Fall 2012

M. D. Jones, Ph.D. (CCR/UB)

Bash Survival Guide for HPC

HPC-I Fall 2012

M. D. Jones, Ph.D. (CCR/UB)

Bash Survival Guide for HPC

HPC-I Fall 2012

M. D. Jones, Ph.D. (CCR/UB)

Bash Survival Guide for HPC

HPC-I Fall 2012

M. D. Jones, Ph.D. (CCR/UB)

Bash Survival Guide for HPC

HPC-I Fall 2012

Is There in Truth No Beauty?

M. D. Jones, Ph.D. (CCR/UB)

Bash Survival Guide for HPC

HPC-I Fall 2012

Note the absence of oating-point operations (serious shortcoming).

M. D. Jones, Ph.D. (CCR/UB)

Bash Survival Guide for HPC

HPC-I Fall 2012

M. D. Jones, Ph.D. (CCR/UB)

Bash Survival Guide for HPC

HPC-I Fall 2012

While loops are just about what you would expect:

M. D. Jones, Ph.D. (CCR/UB)

Bash Survival Guide for HPC

HPC-I Fall 2012

Miscellaneous Advanced Features

M. D. Jones, Ph.D. (CCR/UB)

Bash Survival Guide for HPC

HPC-I Fall 2012

Miscellaneous Advanced Features