8ash Shell Scrlpung for Pellx and

8lowulf
ur. uavld Poover, SC8, Cl1, nlP
hooverdm[hellx.nlh.gov
!anuary 22, 2013
1hls resenLauon Cnllne
http://helix.nih.gov/Documentation/
Talks/BashScripting.tgz
http://helix.nih.gov/Documentation/
Talks/BashScripting.pdf
Culde 1o 1hls resenLauon
1hese llnes are narrauve lnformauon
8elge boxes are lnLeracuve Lermlnal sesslons:
Cray boxes are le/scrlpL conLenLs:
$ echo Today is $(date)
This is the first line of this file.
This is the second line.
Yada yada yada.
!"#$%&' )*+ "*$&%+,-$"%*
8ash
8ash ls a shell, llke 8ourne, korn, and C
Wrluen and developed by Lhe lSl ln 1989
uefaulL shell for mosL Llnux avors
uenluve 8eferences
http://www.gnu.org/software/bash/
manual/
http://www.tldp.org/
http://wiki.bash-hackers.org/
8ash on Pellx and 8lowulf
Pellx runs 8PLL v6, 8ash v4.1.2
8lowulf runs 8PLL v3, 8ash v3.2.23
SubLle dlerences, 3.2.23 ls a subseL of 4.1.2
1hls class deals wlLh 3.2.23
?ou mlghL be uslng 8ash already
$ ssh user@helix.nih.gov
...
Last login: Wed Jan 8 12:27:09 2014 from
123.456.78.90
[user@helix ~]$ echo $SHELL
/bin/bash
[user@helix ~]$ bash
[~]$ bash
lf noL, [usL sLarL a new shell:
?ou should be runnlng bash:
Maybe you're runnlng someLhlng else?
WhaL shell are you runnlng?
$ echo $0
-bash
$ echo $0
-ksh
-csh
-tcsh
-zsh
Shell, kernel, WhaL?
user
keyboard
SPLLL
kernel
hardware (Cu, memory, eLc.)
appllcauons
screen
.##.*$")/#
*nlx rerequlslLe
lL ls essenual LhaL you know someLhlng abouL
unlx or Llnux
lnLroducuon Lo Llnux (Pellx SysLems)
http://helix.nih.gov/Documentation/
#_linuxtut
http://www.unixtutorial.org/
LlemenLal 8ash
8ash ls a command processor: lnLerpreLs
characLers Lyped on Lhe commandllne and
Lells Lhe kernel whaL programs Lo use and how
Lo run Lhem
AkA command llne lnLerpreLer (CLl)
Slmple Command
$ echo Hello World!
prompL command
argumenLs
slmple command
0%$0%,&&" %1 -%22)*+#
Lssenual *nlx Commands
cd ls pwd file wc find du
chmod touch mv cp rm cut paste
sort split cat grep head tail less
more sed awk diff comm ln
mkdir rmdir df pushd popd
date exit ssh rsh printenv time echo
ps jobs [CTRL-c] [CTRL-z] [CTRL-d] top kill
apropos info man help type
Comprehenslve LlsL
hup://hellx.nlh.gov/uocumenLauon/1alks/
8ashScrlpung_LlnuxCommands.pdf
Lxerclse 1:
1ype a command LhaL llsLs all Lhe les ln your /home
dlrecLory and shows le slzes and access permlsslons
Make a dlrecLory, go lnLo lL, and creaLe a le, edlL Lhe
le, Lhen deleLe Lhe le
llnd ouL how much dlsk usage you are uslng
+%-,2.*$)$"%*
uocumenLauon: apropos
apropos ls hlgh level lndex of commands
$ apropos sleep
Time::HiRes(3pm) - High resolution alarm,
sleep, gettimeofday, interval timers
sleep(1) - suspend execution for an
interval of time
sleep(3) - suspend thread execution
for an interval measured in seconds
usleep(3) - suspend thread execution
for an interval measured in microseconds
uocumenLauon: info
info ls a comprehenslve commandllne
documenLauon vlewer
dlsplays varlous documenLauon formaLs
$ info kill
$ info diff
$ info read
$ info strftime
$ info info
uocumenLauon: man
$ man pwd
$ man diff
$ man head
uocumenLauon: type
type ls a builtin LhaL dlsplays Lhe type
of a word
$ type -t rmdir
file
$ type -t if
keyword
$ type -t echo
builtin
$ type -t module
function
$ type -t ls
alias
builtin
8ash has bullL-ln commands (builtin)
echo, exit, hash, printf

uocumenLauon: builtin
uocumenLauon can be seen wlLh help
Speclcs wlLh help [builtin]
$ help
$ help pwd
non-8ash Commands
lound ln /bin, /usr/bin, and
/usr/local/bin
Some overlap beLween 8ash builtin and
exLernal execuLables
$ help time
$ man time
$ help pwd
$ man pwd
#-&"0$#
Why wrlLe a scrlpL?
Cne-llners are noL enough
Culck and dlrLy proLoLypes
MalnLaln llbrary of funcuonal Lools
Clue Lo sLrlng oLher apps LogeLher
Why nC1 Lo use a scrlpL?
!"#$%&' maLh (beyond lnLegers)
!"#$%&' l/C (sockeLs, rsh, Lcp)
!"#$%&' daLa sLrucLures (hashes,
muludlmenslonal arrays)
!"#$%&' le operauons
!"#$%&' pauern maLchlng
WhaL ls ln a ScrlpL?
Commands
varlables
luncuons
Loops
Condluonal sLaLemenLs
CommenLs and documenLauon
Cpuons and semngs
My very llrsL ScrlpL
echo Hello World!
$ cat 'echo Hello World!' > script01.sh
$ cat script01.sh
$ bash script01.sh
Hello World!
$
creaLe a scrlpL and call lL wlLh bash:
LxecuLe WlLh A Shebang!
#!/bin/bash
echo Hello World!
$ chmod +x script01.sh
$ ./script01.sh
lf bash ls Lhe defaulL shell, maklng lL
execuLable allows lL Lo be run dlrecLly

Addlng a shebang specles Lhe scrlpung
language:
uebugglng
Call bash wlLh x v
-x dlsplays commands and Lhelr resulLs
-v dlsplays everyLhlng, even commenLs and
spaces
#!/bin/bash x -v
echo Hello World!
$ bash x v script01.sh
A slmple scrlpL:
varlables slmpllfy and make porLable:
uslng varlables
#!/bin/bash
cd /scratch/Bash_class
mkdir /scratch/Bash_class/test1
cd /scratch/Bash_class/test1
echo 'this is boring' > /scratch/Bash_class/test1/file1
#!/bin/bash
dir=/scratch/Bash_class
cd $dir
mkdir $dir/test1
cd $dir/test1
echo 'this is boring' > $dir/test1/file1
varlables
A ()*+),%& ls consLrucL Lo hold lnformauon,
asslgned Lo a -"*.
varlables are lnlually 01.&21&.
varlables have 34"$&
varlables are a key componenL of Lhe shell's
&1(+*"1#&15
lnLeracuve ScrlpL lnpuL
use Lhe read command Lo lnLeracuvely geL a
llne of lnpuL:
echo -n "Type in a number: "
read response
echo "You said: $response"
$ bash script.sh
Type in a number:
You said: 4
$
Commandllne ScrlpL lnpuL
use Lhe posluonal parameLers Lo lnclude
argumenLs Lo Lhe scrlpL
echo "The first three arguments are:"
echo $1
echo $2
echo $3
$ bash script.sh one two three
The first three arguments are:
one
two
three
$
Lxerclse 2:
CreaLe a scrlpL Lo run a slmple command (for
example echo, ls, date, sleep)
8un Lhe scrlpL
#.$$"*3 $!. .*4"&%*2.*$
LnvlronmenL
A seL of varlables and funcuons recognlzed by
Lhe kernel and used by mosL programs
noL all varlables are envlronmenL varlables,
musL be exported
lnlually seL by 567869: ;<=5
printenv dlsplays varlables and values
SeL a varlable
lormally done uslng declare
very slmple, no spaces
$ declare myvar=100
$ myvar=10
SeL a varlable
Lxamlne value wlLh echo and $:
1hls ls acLually :787>=6=8 =?:7@5AB@
$ echo $myvar
10
export
export ls used Lo seL an envlronmenL
varlable:
?ou can do lL one move
$ export MYENVVAR=10
$ MYENVVAR=10
$ export MYENVVAR
$ printenv MYENVVAR
unset a varlable
unset ls used for Lhls

Can be used for envlronmenL varlables

$ unset myvar
$ echo $HOME
/home/user
$ unset HOME
$ echo HOME

$ export HOME=/home/user
8emove from envlronmenL
export -n
$ declare myvar=10
$ echo $myvar
10
$ printenv myvar # nothing not in environment
$ export myvar
$ printenv myvar # now it's set
10
$ echo $myvar
10
$ export -n myvar
$ printenv myvar # now it's gone
$ echo $myvar
10
8ash varlables
$HOME = /home/[user] = ~
$PWD = currenL worklng dlrecLory
$PATH = llsL of lepaLhs Lo look for
commands
$CDPATH = llsL of lepaLhs Lo look for
dlrecLorles
$TMPDIR = Lemporary dlrecLory (/tmp)
$RANDOM = random number
Many, many oLhers.
printenv
$ printenv
HOSTNAME=helix.nih.gov
TERM=xterm
SHELL=/bin/bash
HISTSIZE=500
SSH_CLIENT=96.231.6.99 52018 22
SSH_TTY=/dev/pts/274
HISTFILESIZE=500
USER=student1
module
$ module avail
module can seL your envlronmenL
Can be used for envlronmenL varlables
$ module load python/2.7.6
$ module unload python/2.7.6
http://helix.nih.gov/Applications/
modules.html
Semng ?our rompL
1he $PS1 varlable (prlmary prompL, $PS2
and $PS3 are for oLher Lhlngs)
Pas lLs own formaL rules
Lxample: PS1=[\u@\h \W]$
[user@host myDir]$ echo Hello World!
Hello World!
[user@host myDir]$
Semng ?our rompL
\d daLe ("1ue May 6")
\h hosLname ("hellx")
\[ number of [obs
\u username
\W basename of $Wu
\a bell characLer (why?)
hup://www.gnu.org/soware/bash/manual/bashref.hLml#rlnung-a-rompL

1esL ScrlpL
module load python/2.7.6
tag=$USER.$RANDOM
mkdir $tag
cd $tag
pip show numpy
variable.sh:
Lxerclse 3:
SeL an envlronmenL varlable
Look aL all your envlronmenL varlables
)/")#.# )*+ 1,*-$"%*#
Allases
A few allases are seL by defaulL

Can be added or deleLed
8emove all allases
$ alias
alias l.='ls -d .* --color=auto'
alias ll='ls -l --color=auto'
alias ls='ls --color=auto
alias vi='vim'
$ unalias ls
$ alias ls='ls CF
$ unalias -a
Allases
Allases belong Lo Lhe shell, buL 678 Lhe
envlronmenL
Allases can 678 be propagaLed Lo subshells
LlmlLed use ln scrlpLs
Cnly useful for lnLeracuve sesslons
Allases WlLhln ScrlpLs
Allases have llmlLed use wlLhln a scrlpL
Allases are noL expanded wlLhln a scrlpL by
defaulL, requlres speclal opuon semng:
Worse, allases are noL expanded wlLhln
condluonal sLaLemenLs, loops, or funcuons
$ shopt -s expand_aliases
luncuons
funcuons are a dened seL of commands
asslgned Lo a word
$ function status() {
> date
> uptime
> who | grep $USER
> checkquota
> }
$ status
Thu Oct 18 14:06:09 EDT 2013
14:06:09 up 51 days, 7:54, 271 users, load average: 1.12, 0.91, 0.86
user pts/128 2013-10-17 10:52 (128.231.77.30)
Mount Used Quota Percent Files Limit
/data: 92.7 GB 100.0 GB 92.72% 233046 6225917
/home: 2.2 GB 8.0 GB 27.48% 5510 n/a
luncuons
dlsplay whaL funcuons are seL:

use declare f Lo dlsplay code
$ declare -F
$ declare -f 'status'
status ()
{
date;
uptime;
who | grep --color $USER;
checkquota
}
luncuons
unset deleLes funcuon

$ unset status
luncuons
funcuons can propagaLe Lo chlld shells uslng
export
$ export -f status
luncuons
local varlables can be seL uslng local
$ export TMPDIR=/tmp
$ function processFile() {
> local TMPDIR=/data/user/tmpdir
> echo $TMPDIR
> sort $1 | grep $2 > $2.out
> }
$ processFile /path/to/file string
/data/user/tmpdir
$ echo $TMPDIR
/tmp
1esL ScrlpL
function throwError
{
echo ERROR: $1
exit 1
}

module load python/2.7.6
tag=$USER.$RANDOM
throwError "No can do!"
cd $tag
pip show numpy
function.sh:
Lxerclse 4:
llnd ouL whaL allases you are currenLly uslng.
uo you llke Lhem? uo you wanL someLhlng
dlerenL?
/%3"*
Logglng ln
ssh ls Lhe defaulL logln cllenL
whaL happens nexL?

$ ssh $USER@helix.nih.gov
ssh logln
bash scrlpL
$8ASP_Lnv
scrlpL process
logouL
l
n
L
e
r
a
c
u
v
e

n
o
n
-
l
o
g
l
n

l
n
L
e
r
a
c
u
v
e

l
o
g
l
n

n
o
n
-
l
n
L
e
r
a
c
u
v
e

bash
~/.bashrc
user sesslon
user sesslon
exlL
b
a
c
k
g
r
o
u
n
d

bash
/eLc/prole
/eLc/prole.d/*
~/.bash_prole
~/.bash_logln
~/.prole
user sesslon
user sesslon
~/.bash_logouL
exlL
b
a
c
k
g
r
o
u
n
d

b
a
c
k
g
r
o
u
n
d

8ash llow
/eLc/bashrc
Logglng ln
lnLeracuve logln shell (ssh from somewhere
else)

/etc/profile
~/.bash_profile

~/.bash_logout (when exlung)
1he sLarLup les are sourced, noL execuLed

source and .
source execuLes a le ln Lhe currenL shell
and preserves changes Lo Lhe envlronmenL
. ls Lhe same as source
~/.bash_profile
$ cat ~/.bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
non-Logln Shell
lnLeracuve non-logln shell (calllng bash from
Lhe commandllne)
8eLalns envlronmenL from logln shell
Shell levels seen wlLh $SHLVL
~/.bashrc
$ echo $SHLVL
1
$ bash
$ echo $SHLVL
2
~/.bashrc
$ cat ~/.bashrc
# .bashrc
if [ -f /etc/bashrc ] ; then
. /etc/bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH


non-lnLeracuve Shell
lrom a scrlpL
8eLalns envlronmenL from logln shell
SeL Lo a le llke ~/.bashrc
$BASH_ENV (lf seL)
$BASH_ENV
1o force modules Lo work ln scrlpLs
$ cat ~/.bash_script_exports
module ()
{
eval `/usr/local/Modules/$MODULE_VERSION/bin/
modulecmd bash $*`
}
export -f module
$ echo export BASH_ENV=~/.bash_script_exports >>
~/.bashrc
ArblLrary SLarLup llle
user-dened (e.g. ~/.my_profile)
Can also use '.'
$ source ~/.my_profile
[dir user]!
$ . ~/.my_profile
[dir user]!
function_depot.sh
A collecuon of useful funcuons
$ cat /scratch/Bash_class/function_depot.sh
1esL ScrlpL
module load python/2.7.6
source function_depot.sh
tag=$USER.$RANDOM
throwError "No can do!"
cd $tag
pip show numpy
function2.sh:
#"20/. -%22)*+#
uenluons
command arg1 arg2 .
slmple command 1 | slmple command 2 |& .
plpellne 1 , plpellne 2 , .
Slmple command /
process
lpellne / !ob
LlsL
word
Command
WhaL ls Lhe currenL ume and daLe?
Where are you?
Some command examples
$ date
$ pwd
Slmple Command
$ echo Hello World!
prompL command
argumenLs
slmple command
LlsL Lhe conLenLs of your /home dlrecLory
Pow much dlsk space are you uslng?
WhaL are you up Lo?
Slmple commands
$ ls -l -a $HOME
$ du -h -s $HOME
$ ps -u $USER -o size,pcpu,etime,comm --forest
funcuon?
bullun?
ln $A1P
Lxec le?
LxecuLe
command
Slashes? Command
yes
yes
yes
yes no
no
no
no
yes
L88C8
no
Command Search 1ree
Allas
expanslon
Word
expanslon
$ cmd arg1 arg2
hlL reLurn
rocess
A $*"4&33 ls an execuung lnsLance of a slmple
command
Can be seen uslng ps command
Pas a unlque ld ($*"4&33 +., or $+.)
8elongs Lo a process group
top command
ps command
wazzup
wazzup and watch
wazzup ls a wrapper scrlpL for
watch repeaLs a command every 2 seconds
useful for Lesung appllcauons and meLhods
$ ps -u you --forest --width=2000 -o
ppid,pid,user,etime,pcpu,pmem,{args,comm}
"*0,$ )*+ %,$0,$
8edlrecuon
Lvery process has Lhree le descrlpLors (le
handles): S1uln (0), S1uCu1 (1), S1uL88 (2)
ConLenL can be redlrecLed
cmd > x.out 8edlrecL le descrlpLor 1 from S1uCu1 Lo x.ouL
cmd < x.in 8edlrecL le descrlpLor 0 from S1uln Lo x.ln
cmd 1> x.out 2> x.err
8edlrecL le descrlpLor 1 from S1uCu1 Lo x.ouL,
le descrlpLor 2 from S1uL88 Lo x.err
Comblne S1uCu1 and S1uL88

Crderlng ls lmporLanL
cmd > x.out 2>&1
8edlrecL le descrlpLor 1 from S1uCu1 Lo lename
x.ouL, Lhen redlrecL le descrlpLor 2 from S1uL88 Lo
wherever le descrlpLor 1 ls polnung (x.ouL)
cmd 2>&1 > x.out 8edlrecL le descrlpLor 2 from S1uL88 Lo wherever le
descrlpLor 1 ls polnung (S1uCu1), Lhen redlrecL le
descrlpLor 1 from S1uCu1 Lo lename x.ouL
CorrecL:
lncorrecL:
cmd 2>&1
8edlrecL le descrlpLor 2 from S1uL88 Lo wherever le
descrlpLor 1 ls polnung (S1uCu1)
8edlrecuon Lo a llle
use beuer synLax lnsLead - Lhese all do Lhe
same Lhlng:
cmd > x.out 2>&1
cmd 1> x.out 2> x.out
cmd &> x.out
W8CnC!
cmd >& x.out
8edlrecuon
Appendlng Lo a le
cmd >> x.out
Append S1uCu1 Lo x.ouL
Comblne S1uCu1 and S1uL88, append Lo x.ouL
cmd 1>> x.out 2>&1
cmd >>& x.out W8CnC!
cmd &>> x.out Cnly avallable on bash v4 and above (Pellx, noL 8lowulf)
named lpes/lllC
Send S1uCu1 and/or S1uL88 lnLo Lemporary
le for commands LhaL can'L accepL ordlnary
plpes
$ ls /home/$USER > file1_out
$ ls /home/$USER/.snapshot/Weekly.2012-12-30* >
file2_out
$ diff file1_out file2_out > diff.out
$ rm file1_out file2_out
$ cat diff.out
named lpes/lllC
lllC speclal le can slmpllfy Lhls
?ou Lyplcally need muluple sesslons or shells
Lo use named plpes
[sess1]$ mkfifo pipe1
[sess1]$ ls /home/$USER > pipe1
[sess2]$ cat pipe1
rocess SubsuLuuon
1he operaLors <( ) and >() can be used Lo
creaLe LranslenL named plpes
AkA process subsuLuuon
1he operaLors '> >()' are equlvalenL Lo
'|'
$ diff <(ls /home/$USER) <(ls /home/$USER/.snapshot/
Weekly.2012-12-30*)
$ cat /usr/share/dict/words > >(grep zag)
1esL ScrlpL
diff <(zcat webfiles.new.gz > >(sort) ) <(zcat
webfiles.old.gz > >(sort) )
fifo.sh:
Lxerclse 3:
8un a command and redlrecL Lhe STDOUT
lnLo a new le.
8edlrecL Lhe STDOUT lnLo a le and Lhe
STDERR lnLo /dev/null
0"0./"*.# )*+ C%D#
lpellne
STDOUT of each slmple command ls passed
as STDIN of Lhe nexL command
$ ps -ef | grep sh | head -3
lpellne
STDERR can be comblned wlLh STDOUT
$ ls emptydir | grep -c 'No such file'
ls: cannot access emptydir: No such file or
directory
0
$ ls emptydir 2>&1 | grep -c 'No such file'
1
1esL ScrlpL
cat <(cut -d',' -f1,2,22-33 genome_stuff.csv | \
head -1 | tr ',' $'\t') <(cut -d',' \
-f1,2,22-33 genome_stuff.csv \
| grep chrX | tr ',' $'\t' | sort -k3)
pipes.sh:
1esL ScrlpL
cat <(zgrep ^chrX ins.bed.gz | \
awk '{if ($2 > 100000000 && $2 < 100500000) print "INS
\t" $0}') \
<(zgrep ^chrX del.bed.gz | \
awk '{if ($2 > 100000000 && $2 < 100500000) print "DEL
\t" $0}') \
<(zgrep ^chrX jun.bed.gz | \
awk '{if ($2 > 100000000 && $2 < 100500000) print "JUN
\t" $0}') | \
sort -nk3,4
pipes2.sh:
!ob ConLrol
A 9", ls anoLher name for plpellne
Lach slmple command ls a process
$ echo Hello World! > x | cat x | grep o
loreground and 8ackground
A [ob (plpellne) runs ln Lhe foreground by
defaulL
:3;14<*"1"03 [obs are run ln background (ln
parallel, re and forgeL)
8equlres slngle & aL Lhe end
8ackground [obs run ln Lhelr own shell
LxlL sLaLus ls noL avallable
$ sleep 5 &
wait
1he wait command does [usL LhaL
useful for consolldauon or summary sLeps
$ sleep 9 &
$ sleep 7 &
$ sleep 5 &
$ wait
Common roblem
A background process wlLhouL redlrecuon wlll
dle lf Lhe Lermlnal ls dlsconnecLed and Lhe
process Lrles Lo wrlLe Lo STDIN or STDOUT
Make sure all background processes redlrecL
STDIN and STDOUT
nohup does Lhe same Lhlng
$ STDIN_FAIL &
$ STDIN_FAIL &> junk.out < /dev/null &
$ nohup STDIN_FAIL &
!ob ConLrol
1he shell lLemlzes [obs by number
$ sleep 10 &
[1] 8683
$ sleep 10 &
[2] 8684
$ sleep 10 &
[3] 8686
$ jobs
[1] Running sleep 10 &
[2]- Running sleep 10 &
[3]+ Running sleep 10 &
$
[1] Done sleep 10
[2]- Done sleep 10
[3]+ Done sleep 10
!ob ConLrol
A [ob can be moved from foreground Lo
background wlLh ctrl-z and bg
Can be broughL back wlLh fg
$ step1.sh | step2.sh | grep normal
ctrl-z
[1]+ Stopped step1.sh | step2.sh ..
bg
[1]+ step1.sh | step2.sh ..
$ jobs
[1]+ Running step1.sh | step2.. &
$ fg
step1.sh step2.sh ...
Lxerclse 6:
uo an ls on a dlrecLory, plpe Lhe resulLs Lo
grep, and Lhen sort Lhe resulLs.
-%22)*+ /"#$#
Command LlsL
Sequence of one or more [obs separaLed by
;, &, &&, or ||.
Slmple commands/plpellnes/[obs are run
sequenually when separaLed by ;
$ date ; sleep 5 ; sleep 5 ; sleep 5 ; date
$ echo 1 > x ; echo 2 > y ; echo 3 > z
Command LlsL
Sequenual command llsL ls equlvalenL Lo
or
$ echo 1 > x
$ echo 2 > y
$ echo 3 > z
$ date
$ sleep 5
$ sleep 5
$ sleep 5
$ date
Command LlsL
Asynchronously when separaLed by &
wait pauses unul all background [obs
compleLe
$ date ; sleep 5 & sleep 5 & sleep 5 & wait ; date
$ echo 1 > x & echo 2 > y & echo 3 > z &
Command LlsL
Asynchronous command llsL ls equlvalenL Lo
or
$ echo 1 > x &
$ echo 2 > y &
$ echo 3 > z &
$ date
$ sleep 5 &
$ sleep 5 &
$ sleep 5 &
$ wait
$ date
Crouped Command LlsL
1o execuLe a llsL of sequenual plpellnes ln Lhe
background, or Lo pool STDOUT/STDERR,
enclose wlLh () or {}
STDIN ls nC1 pooled, buL musL be redlrecLed
wlLh each command or plpellne.
$ ( cmd 1 < input ; cmd 2 < input ) > output &
[1] 12345
$ { cmd 1 < input ; cmd 2 < input ; } > output &
[2] 12346
Crouped Command LlsL
'{ }' runs ln Lhe currenL shell
'( )' runs ln a chlld/sub shell
Crouped Command LlsL ueLalls
$ (sleep 3 ; sleep 5 ; sleep 8)
user 6299 6283 0 09:29 ? 00:00:00 sshd: user@pts/170
user 6303 6299 0 09:29 pts/170 00:00:00 \_ -bash
user 5040 6303 0 16:33 pts/170 00:00:00 \_ -bash
user 5100 5040 0 16:33 pts/170 00:00:00 \_ sleep 8
$ { sleep 3 ; sleep 5 ; sleep 8 ; }
user 6299 6283 0 09:29 ? 00:00:00 sshd: user@pts/170
user 6303 6299 0 09:29 pts/170 00:00:00 \_ -bash
user 6980 6303 0 16:35 pts/170 00:00:00 \_ sleep 8
Crouped Command LlsL ueLalls
Crouped command llsL run ln background are
ldenucal Lo '( )'
$ (sleep 3 ; sleep 5 ; sleep 8) &
$ { sleep 3 ; sleep 5 ; sleep 8 ; } &
user 6299 6283 0 09:29 ? 00:00:00 sshd: user@pts/170
user 6303 6299 0 09:29 pts/170 00:00:00 \_ -bash
user 17643 6303 0 16:50 pts/170 00:00:00 \_ -bash
user 17696 17643 0 16:50 pts/170 00:00:00 \_ sleep 8
Crouped Command LlsL ueLalls
$ (sleep 3 & sleep 5 & sleep 8 &)
user 6299 6283 0 09:29 ? 00:00:00 sshd: user@pts/170
user 6303 6299 0 09:29 pts/170 00:00:00 \_ -bash
user 7891 1 0 16:37 pts/170 00:00:00 sleep 8
user 7890 1 0 16:37 pts/170 00:00:00 sleep 5
user 7889 1 0 16:37 pts/170 00:00:00 sleep 3
$ { sleep 3 & sleep 5 & sleep 8 & }
user 6299 6283 0 09:29 ? 00:00:00 sshd: user@pts/170
user 6303 6299 0 09:29 pts/170 00:00:00 \_ -bash
user 9165 6303 0 16:39 pts/170 00:00:00 \_ sleep 3
user 9166 6303 0 16:39 pts/170 00:00:00 \_ sleep 5
user 9167 6303 0 16:39 pts/170 00:00:00 \_ sleep 8
1esL ScrlpL
tag=$RANDOM
zcat geography.txt.gz > $tag.input

function reformat {
cat <(grep "^$1" $tag.input | sort -nk2) <(echo "--------") >
$tag.file$2
}

reformat "Africa" 1
reformat "Asia" 2
reformat "Europe" 3
reformat "North America" 4
reformat "Oceania" 5
reformat "South America" 6

cat $tag.file*
rm $tag.file*
geography_{serial,parallel}.sh:
#,D#!.//#
Shells and Subshells
A shell ls dened by an envlronmenL and a
currenL worklng dlrecLory (CWu)
Subshells (chlld shells) lnherlL Lhe
envlronmenL and CWu
Changes Lo Lhe envlronmenL and CWu are !"#
propagaLed back Lo Lhe parenL
1alklng 1o ?ourself
$ pwd
/home/user
$ export NUMBER=5
$ bash
$ ((NUMBER++))
$ echo $NUMBER
6
$ cd /scratch
$ exit
$ echo $NUMBER
5
$ pwd
/home/user
A@ 7 59E5F=<<
1esL ScrlpL
pwd
for i in {1..10} ; do
(
BASE=/scratch/$i.$RANDOM
mkdir $BASE
cd $BASE
pwd
sleep 5
rmdir $BASE
)
done
pwd
subshell.sh:
Lxerclse 7:
WrlLe a scrlpL LhaL runs muluple lnsLances of
subshell.sh ln Lhe background, and
waits for Lhe resulL
use monlLorlng Lools (top, ps, wazzup)
Lo waLch whaL happens
-%22)*+ /"*. "*$.&0&.$.&
WaLch WhaL ?ou WrlLe
8ash lnLerpreLs whaL you wrlLe )=&* you hlL
reLurn
auerns of characLers can cause =?:7@5AB@
arameLer Lxpanslon
$ ls placed ouLslde for parameLer expanslon
8races can be used Lo preserve varlable
$ name=monster
$ echo $name
monster
$ echo $name_silly

$ echo ${name}_silly
monster_silly
arameLer Lxpanslon

More Lhan one ln a row
$ var1=cookie
$ var3=_is_silly
$ echo ${var1}_${name}${var3}
cookie_monster_is_silly
8race Lxpanslons
8race expanslon { , , }
8race expanslon { .. }

$ echo {bilbo,frodo,gandalf}
bilbo frodo gandalf
$ echo {0,1,2,3,4,5,6,7,8,9}
0 1 2 3 4 5 6 7 8 9
$ echo {0..9}
0 1 2 3 4 5 6 7 8 9
$ echo {bilbo..gandalf}
{bilbo..gandalf}
$ echo {b..g}
b c d e f g
8race Lxpanslons
nesLed brace expanslons

ulsuncL from parameLer expanslon $ or ${
$ mkdir z{0,1,2,3,4,5,6,7,8,9}
$ ls
z0 z1 z2 z3 z4 z5 z6 z7 z8 z9
$ rmdir z{{1..4},7,8}
$ ls
z0 z5 z6 z9
$ echo {$var1,${name},brought,to,you,by,{1..3}}
cookie monster brought to you by 1 2 3
ArlLhmeuc Lxpanslon
(( )) ls used Lo evaluaLe maLh
$ ls placed ouLslde for parameLer expanslon
$ echo ((12-7))

$ echo $((12-7))
5
ArlLhmeuc Lxpanslon
varlables can be updaLed, noL [usL evaluaLed
$ a=4
$ b=8
$ echo $((a+b))
12
$ echo $a
4
$ echo $b
8
$ echo $((a=a+b))
12
$ echo $a
12
ArlLhmeuc Lxpanslon
1he ++ and -- operaLors only work on
varlables, and updaLe Lhe value
$ a=4
$ ((a++))
$ echo $a
5
$ unset b
$ ((b--))
$ echo $b
-1
$ ((4++))
-bash: 4++: syntax error: operand expected (error token is
"+")
ArlLhmeuc - lnLegers only
8ash can only handle lnLegers

$ a=4.5
$ ((a=a/3))
-bash: ((: 4.5: syntax error: invalid arithmetic operator
(error token is ".5")
ArlLhmeuc - lnLegers only
8ash can only do lnLeger maLh
ulvlslon by zero ls caughL wlLh exlL sLaLus
$ a=3
$ ((a=a/7))
$ echo $a
0
$ ((a=a/0))
-bash: let: a=a/0: division by 0 (error token is "0")
ArlLhmeuc Lxpanslon
MaLh ls done uslng let and (( ))
$ a=1
$ echo $a
1
$ let a++
$ echo $a
2
$ ((a++))
$ echo $a
3
$ let a=a+4
$ echo $a
7
Command SubsuLuuon
` = backuck
beuer Lo use $()
nesLed processes
$ echo uname -n
uname n
$ echo `uname -n`
biowulf.nih.gov
$ echo $(uname -n)
biowulf.nih.gov
$ y=$(du -hs $(ls -1))
1ab Lxpanslon
$ ls /usr/
bin/ lib/ local/ sbin/ share/
$ ls /usr/

<+5 5), <&*&
1llde Lxpanslon
$ echo ~
/home/user
auern MaLchlng
* : maLch any sLrlng
? : maLch any slngle characLer
[?-?] : maLch range of characLers
[!?] or [^?] : noL maLch characLer
$ touch {{1..9},{a..z}}
$ ls [a-e1-4]
1 2 3 4 a b c d e
$
CharacLer Classes
[[:CLASS:]] can be lncluded wlLh pauern
maLchlng
full llsL
$ touch {1..9} y{1..9} z{1..9}
$ ls [[:alpha:]]4
y4 z4
alnum cntrl print word
alpha digit punct xdigit
ascii graph space
blank lower upper
CharacLer Classes
uslng [[:CLASS:]]
$ touch %5 A3 x.9
$ ls [[:upper:]]*
A3
$ ls *[[:punct:]]*
%5 x.9
$ ls [![:lower:]]*
%5 A3
CuoLes
Slngle quoLes preserve llLeral values
uouble quoLes allow varlable and shell
expanslons
$ echo cd $PWD $(uname -n)
cd $PWD $(uname -n)
$ echo cd $PWD $(uname -n)
cd /home/user helix.nih.gov
CuoLes
uouble quoLes also preserve blank characLers
use $'\t' Lo lnserL a Lab
use $'\n' Lo lnserL a newllne

$ msg=$(echo Hi$'\t'there.$'\n'How are you?)
$ echo $msg
Hi there. How are you?
$ echo "$msg"
Hi there.
How are you?
5),> 1"5 3$)4&
Lscapes
1he escape characLer \ preserves llLeral
value of followlng characLer
Powever, lL LreaLs newllne as llne conunuauon
$ echo \$PWD is $PWD
$PWD is /home/user
$ echo \$PWD is $PWD \
> something else
$PWD is /home/user something else
Lscapes
lunny non-prlnung characLers
$'char'
$ echo Hello World
Hello World
$ echo $'\n\n'Hello$'\t'World$'\n\n'


Hello World

Lxerclse 8:
WrlLe a scrlpL LhaL renames a seL of les wlLh
Lhe same basename, buL dlerenL sumxes
uslng a varlable for Lhe basename and
parameLer expanslon for Lhe sumxes
Lxamples:
myle.bam, myle.bam.bal, myle.bam.bas
myle.pdb, myle.psf, myle.dcd, myle.coor
-%*+"$"%*)/ #$)$.2.*$#
LxlL SLaLus
A process reLurns an exlL sLaLus (0-233)
0 = success (almosL always)
1 = general error, 2-233 = speclc error
SLored ln $? parameLer
$ cat /var/audit
cat: /var/audit: Permission denied
$ echo $?
1
$ ls /zzz
ls: cannot access /zzz: No such file or directory
$ echo $?
2
if
$ if (( 1 == 1 ))
> then
> echo yes
> fi
yes
$ if [[ -f myfile ]]
> then echo exists
> fi
$ if ls; then echo $'\n' Done! $'\n'; fi
file1 file2 file3

Done!

$
Pow Many Cores or 1hreads
#!/bin/bash
i=$(sort -u /proc/cpuinfo | grep -c '^physical id')
c=$(sort -u /proc/cpuinfo | grep -c '^core id')
t=$(sort -u /proc/cpuinfo | grep -c '^processor')

if [[ $i != 0 ]] ; then cpu=$((i*c)) ; else cpu=$t ; fi

echo cpu = $cpu
echo threads = $t
if .. elif .. else .. fi
if test-commands
then consequent-commands
elif more-test-commands
then more-consequents
else alternate-consequents
fi
if .. elif .. else .. fi
if test-commands ; then
consequent-commands
elif more-test-commands ; then
more-consequents
else
alternate-consequents
fi
Condluonals - if
lf .. Lhen .. ellf .. else ..
[ ls a builtin, equlvalenL Lo test
very slmllar Lo condluonal llsL

[[ ]] ls exLended test command
$ if [ -e file ] ; then echo file exists ; fi
$ if test -e file ; then echo file exists ; fi
$ [ -e output ] && echo file exists
$ [[ -e output ]] && echo file exists
Condluonals - if
MusL be a space beLween LesL sLaLemenL and
brackeLs
$ if [[-e file]] ; then echo file exists ; fi
-bash: [[-e: command not found
$ if [[ -e file]] ; then echo file exists; fi
-bash: syntax error in conditional expression: unexpected
token `;'
-bash: syntax error near `;'
8ooleans for MaLh
Can use maLh wlLh condluonals ln muluple
LesLs
$ a=4
$ if ((a==4)) ; then echo yes ; else echo no ; fi
yes
$
$ if (( (a-5) == 0 )) ; then echo yes ; else echo no ; fi
no
$ if (( a < 10 )) ; then echo yes; else echo no ; fi

8ooleans for MaLh
8e careful wlLh equal slgns!
$ a=4
$ echo $a
4
$ if (( a = 5 )) ; then echo yes ; else echo no ; fi
yes
$ echo $a
5
1esL scrlpL
matching.sh
echo -n "Type in something : "
read response
echo "You said: $response"

if [[ "$response" == [0-9][0-9][0-9] ]] ; then
echo "This is a three-digit number!"

elif $( echo "$response" | grep -q "^[0-9]\+$" ) ; then
echo "This is a integer!"

elif $( echo "$response" | grep -q "^[0-9]\+\.[0-9]*$" ) ; then
echo "This is a floating point number!"

elif $( echo "$response" | egrep -q '^(\+|-)?[0-9]+\.?[0-9]+(e\
+[0-9]+|e-[0-9]+|e[0-9]+)?$' ) ; then
echo "This is scientific notation!"
fi
Condluonals: test and [[ ]]
test has many dlerenL prlmarles and
operaLors
Can be used for les (dlrecLorles) or
comparlng sLrlngs and varlables
8oLh test and [[ ]] are bulluns, [ ls a
program, and test ls also a program
$ help test
$ help [[
$ man test
$ man [[
auern MaLchlng Condluonals
1esL lf a sLrlng maLches a pauern
ls a varlable a number?
uoes a sLrlng end wlLh an alphabeucal
characLer?
$ a=939
$ [[ $a =~ "^[0-9]+$" ]] && echo is a number
is a number
$ a=939
$ if [[ $a == *[[:alpha:]] ]] ; then echo yes ; else echo
no ; fi
no
Condluonals and exlL sLaLus
if and test use Lhe exlL sLaLus of Lhe
evaluaLed sLaLemenL
$ test -d /tmp
$ echo $?
0
$ test -f silly
$ echo $?
1
$ [[ -s nonsense ]]
$ echo $?
1
Ceneral if sLaLemenLs
if evaluaLes exlL sLaLus, so lL can be used
wlLh any command
1he lnverse ls posslble wlLh !
ldenucal Lo grouped command llsL
$ if grep -q bashrc ~/.bash_profile ; then echo yes ; fi
$ if ! cat /zzz &> /dev/null ; then echo empty ; fi
$ [[ -e /tmp ]] && echo "/tmp exists!"
/tmp exists!
$ [[ $PATH == $HOME ]] || echo "not equal"
not equal
1esL scrlpL
stepping.sh
iterations=5
iterations_remaining=$iterations
seconds_per_step=2
while (( $iterations_remaining > 0 ))
do
if (( ($(date +%s) % $seconds_per_step) == 0 ))
then
echo -n "$iterations_remaining : "
( date ; sleep 10 ) &
((iterations_remaining--))
fi
done
Condluonal Command LlsL
A llsL can execuLe sequence plpellnes
condluonally
LxecuLe cmd2 lf cmd1 was successful
LxecuLe cmd2 lf cmd1 was 678 successful
Condluonal llsLs can be grouped uslng slngle
parenLheses:
$ run_true && run_false
$ run_false || run_true
$ ( run_random || ( run_true && run_false ) )
8oolean CperaLors
Muluple if sLaLemenLs ln serles
$ if [ -e file1 ] && [ -e file2 ] ; then echo both ; fi
case .. esac
case word in
pattern)
consequent-commands ;;
more patterns)
more-test-commands ;;
esac
Condluonals -case
date reLurns Lhe currenL daLe
date +%someLhlng changes Lhe formaL
date +%A reLurns full day of Lhe week (e.g.
Wednesday)
date +%a reLurns abbrevlaLed uC1W (e.g.
Wed)
Condluonals - case
case .. esac uses pauern maLchlng
$ case $(date +%a) in
Mon | T?? | Wed | Fri ) echo "Weekday" ;; \
S* ) echo "Weekend" ;; \
*) echo Something Else ;; \
esac
Weekday
$
$ case $(date +%A) in
Mon | T?? | Wed | Fri ) echo "Weekday" ;; \
S* ) echo "Weekend" ;; \
*) echo Something Else ;; \
esac
Something Else
Lxerclse 9:
WrlLe a scrlpL LhaL auempLs Lo read Lhe lasL 10
llnes of a le lnLo a varlable. lnclude
condluonal sLaLemenLs LhaL LesL for exlsLence,
read access, and number of llnes before
dlsplaylng Lhe value of Lhe varlable.
)&&)'#
WhaL ls an array
An array ls a llnear, ordered seL of values
1he values are lndexed by lnLegers
value
0
value value value
1 2 3
Arrays
Arrays are lndexed by lnLegers (0,1,2,.)
Arrays are referenced wlLh {} and []
$ array=(apple pear fig)
$ echo ${array[*]}
apple pear fig
$ echo ${array[2]}
fig
$ echo ${#array[*]}
3
Arrays vs. varlables
Arrays are acLually [usL an exLenslon of
varlables
$ var1=apple
$ echo $var1
apple
$ echo ${var1[0]}
apple
$ echo ${var1[1]}

$ echo ${#var1[*]}
1
$ var1[1]=pear
$ echo ${#var1[*]}
2
Arrays and Loops
Arrays are very helpful wlLh loops
$ for i in ${array[*]} ; do echo $i ; done
apple
pear
fig
uslng Arrays
1he number of elemenLs ln an array
Walk an array by lndex
$ num=${#array[@]}
$ for (( i=0 ; i < ${#array[@]} ; i++ ))
> do
> echo $i: ${array[$i]}
> done
0: apple
1: pear
2: fig
1esL scrlpL
arrays.sh
declare -a array
array=(apple pear fig)

echo There are ${#array[*]} elements in the array

echo Here are all the elements in one line: ${array
[*]}

echo Here are the elements with their indices:

for (( i=0 ; i < ${#array[@]} ; i++ ))
do
echo " $i: ${array[$i]}"
done
/%%0#
for .. do .. done
for name in words
do
commands
done
for (( expr1 ; expr2 ; expr3 ))
do
commands
done
Loops - for
for ls used Lo sLep Lhrough muluple words
lmlLaLes until and while uslng seq:
$ for i in $(seq 1 5) ; do echo $i ; done
1
2
3
4
5
$ for i in apple pear fig ; do echo $i ; done
apple
pear
fig
genome_nonsense.sh
#!/bin/bash
module load R/2.14
for i in {1..22} X Y M ; do
label=$i
if [[ $i == [[:digit:]] ]]; then
label=$(printf '%02d' $i)
fi
[[ -f x/$label/trial.out ]] && break
[[ -d x/$label ]] || mkdir -p x/$label
pushd x/$label 2>&1 > /dev/null
echo Running chr${i}_out
# actually do something, not this
touch trial_chr${i}.out
popd >& /dev/null
done
C-sLyle for loop
for can be used for lnLeger Lraversal
$ for (( i=0 ; i < 10 ; i++ ))
> do
> echo $i
> done
0
1
2
3
4
5
6
7
8
9
while .. do .. done
while test-commands
do
consequent-commands
done
until .. do .. done
until test-commands
do
consequent-commands
done
Loops - until
unul uses test commands
Pandy wlLh maLh
Can be used wlLh semaphore les
$ a=1 ; until [[ $a -gt 5 ]] ; do echo $a ; let a++ ;
done
1
2
3
4
5
$ until [[ -e stop.file ]] ; do sleep 60 ; done
Loops - while
while ls Lhe reverse of until
$ a=1 ; while [[ $a -le 5 ]] ; do echo $a ; let a++ ;
done
1
2
3
4
5
break And continue
Can be used Lo end loops or sklp secuons
$ while [[ $a -le 5 ]] ; do
> echo $a
> if (( a == 3 )) ; then break ; fi
> let a++
> done
0
1
2
3
Using while and read in a
script

while read var
do
if [[ $var == "exit" ]]
then
break
fi
echo $var
# do something else with $var
done
readfile.sh:
$ bash readfile.sh < genome_stuff.csv
uslng for and case ln a scrlpL

arg=($@)
for (( i=0 ; i < ${#arg[@]} ; i++ ))
do
let j=i+1
case ${arg[$i]} in
--help) halp=1 ;;
--nonsense) nonsense=1 ;;
--extra-juicy) extra=1 ;;
esac
done

[[ $halp ]] && echo "Help? What help?" && exit
echo "Now for something completely different"
options.sh:
Lxerclse 10:
WrlLe a scrlpL LhaL does someLhlng 10 umes
LxLra credlL: modlfy Lhe loop Lo qulL lf Lhe
scrlpL spends more Lhan 10 seconds ln Lhe
loop (noL valld lf Lhe execuLed sLeps run very
qulckly, of course)
#$,0"+ 0.$ $&"-G#
LxLended MaLh
use bc lnsLead:
MlghL as well use perl or pyLhon lnsLead.
$ number=$(echo scale=4; 17/7 | bc)
$ echo $number
2.4285
$ x=8
$ y=3
$ z=$(echo "scale = 3; $x/$y" | bc)
$ echo $z
2.666
grep
llnd only LhaL whlch maLched
8ecurslve search
use a llsL of maLches from a le
$ grep -o "Jan *[[:digit:]] ..:.."
$ grep -l Bash_Class *
$ grep -f complex.grep -h *.txt
egrep
LxLended grep allows addluonal
meLacharacLers llke +, ?, |, {} and ()
{} maLch number of umes
$ egrep ftp/data/[^\/]+/exome_alignment/[^\/]+\.mapped
\.ILLUMINA\.bwa\.[^\/]+\.exome\.[^\/]+\.bam
$ ls -l | egrep -o '[A-Z][a-z]{2} [0-9]{1,2} [0-9]{4}'
find
llnd large les
llnd recenLly modled les
$ find . -type f -size +1G
$ find [path] -mmin -10
find
llnd and updaLe les and dlrecLorles ln a
shared dlrecLory

$ find . -user [you] ! -group [grp]
$ find . -user [you] ! -group [grp] -exec chgrp
[yourgroup] {} \;

$ find . -user [you] -perm /u+r ! -perm /g+r
$ find . -user [you] -perm /u+r ! -perm /g+r -exec chmod g
+r {} \;

$ find . -user [you] -perm /u+x ! -perm /g+x
$ find . -user [you] -perm /u+x ! -perm /g+x -exec chmod g
+x {} \;
find
llnd newesL and oldesL les ln a dlrecLory Lree
llnd blggesL and smallesL les
llnd empLy dlrecLorles

$ find . -type f -printf '%T@ %Tc %p\n' | sort -nr | head
$ find . -type f -printf '%T@ %Tc %p\n' | sort -nr | tail
$ find . -type f -printf '%s %Tc %p\n' | sort -nr | head
$ find . -type f -printf '%s %Tc %p\n' | sort -nr | tail
$ find . -type d -empty -printf "%Tc %p\n"
sorL
SorL llnes of LexL by columns of dlerenL Lypes
SorL a very large le wlLh a memory llmlL
$ ls -l /home | sort -k6M -k7n -k8
$ sort huge_tab_delimited_file.txt S 25%
Spaces ln le names
8eplace newllne wlLh null characLer
use IFS= and double quoLes
$ ls -1 | tr '\n' '\0' | xargs -0 du hs
$ find . -type f -print0 | xargs -0 wc
$ while IFS= read -r line; do du -hs "$line"; done < <(ls
-1)
1ab dellmlLed les
8earrange order of columns uslng awk
8earrange order uslng while read
$ awk 'BEGIN { FS = "\t" } ; {print $3,$2,$1}' file3
$ while IFS=$'\t' read -r field1 field2 field3 others
> do
> echo ${field2}$\t${field3}$\t${field1}
> done < /path/to/file
awk and sed
awk ls a scrlpung language
sed ls a le/sLream manlpulauon language
$ awk {sum += $3} END {print sum/NR}
$ sed s/Old line from file/New improved line/g
myfile.txt
$ sed -n 100,120p myfile.txt
1)*$)#' H%&G1/%H #-&"0$
8ash Scrlpung 1arball
uownload from:
hup://hellx.nlh.gov/uocumenLauon/1alks/8ashScrlpung.Lgz
$ ls -1
Empty_Raw_Data.txt
Proper_Unix_File.txt
Stupid Windows Or Mac Name.txt
cmdA
cmdB
cmdC
fantasy.sh
$ ./fantasy.sh rawdata name
8aw
daLa
cmdA
leA.ouL
llleA.zzz
leA.log
cmd8
u8
llle
le8.ouL
SorL
lllLer
8eformaL
leC.ln
cmdC
resulLs
8esulLs u8
May fall!
May be empLy!
May be empLy!
May overwrlLe
prevlous resulLs!
needs large memory!
need loLs of dlskspace!
Append only!
lndeLermlnaLe
number of
resulLs!
8equlres envlronmenL varlable!
needs exLra
$A1P!
1akes a long ume!
MusL have unlque name!
le8.err
May have errors!
MusL exlsL!
.I$&)#
Lxpanded LlsL of Llnux Commands
a2p chmod dos2unlx geLopLs lp openssl readllnk sorL Lree wc
a2ps chown du ghosLscrlpL lpgeL passwd readonly source Lrue wgeL
acroread chsh echo glxlnfo ln pasLe rename spllL LseL whaus
allas cksum egrep gpg local paLch renlce spllLdl Lype wherels
apropos clear emacs grep locaLe paLhchk reseL sqllLe3 ullmlL whlch
aL cmp enable groups lockle pdf2ps reslze ssh umask whlpLall
awk column enscrlpL gs logln perl reLurn sLaL unallas who
basename comblnedl env gunzlp logname pgrep rev sLrace uname whoaml
bash comm eval gvlm logouL php rexec sLream unexpand whols
baLch conunue exec gvlmdl look plco rlogln sLrlngs unlq xargs
baLchllm cp exlL gzlp ls pldof rm submlunfo unlx2dos xclock
bc cronLab expand hash lsof plng rmdlr suspend unllnk xdl
bg csh exporL head mac2unlx plnky rsh svn unseL xeyes
blnd cspllL expr help man pklll rsync swarm unzlp xLerm
break curl facLor hlsLory man2hLml popd scp swarmdel updaLedb yes
bunzlp2 cuL false hosL md3sum pr screen Lac upume zcaL
bzcaL cvs fg hosLname merge prlnLenv scrlpL Lall urlvlew zcmp
bzcmp daLe fgrep lconv mkdlr prln sdl Lallf users zdl
bzdl dc le ld mkfo ps sed Lar usleep zforce
bzgrep dd nd lnfo more pushd sendmall Lcsh vdlr zgrep
bzlp2 declare nger lnLerdl mv pwd seq Lee vl zlp
bzlp2recover df ock losLaL namel pyLhon seL LelneL vlew zlpgrep
bzless dl fmL [obcheck nano qdel seLsld LesL vlm zlplnfo
bzmore dl3 fold [obload neLsLaL qselecL sp ume vlmdl zlpnoLe
cal dlr free [obs newer qsLaL sg umeouL3 vlmLuLor zlpspllL
caL dlrcolors freen [oln newgrp qsub sh umes vmsLaL zless
cd dlrname p klll nlce quoLa shl Lop zmore zsh
checkquoLa dlrs funzlp ksh nl rcp shopL Louch walL
chfn dlsown gawk less nohup rdlsL shred Lr wall
chgrp dlsplay gedlL leL oclock read sleep Lrap waLch
uenluons
CharacLer: resulL of
a keysLroke
MeLacharacLer:
separaLes words
8lank: space, Lab,
or newllne
characLer
meLacharacLer

| & , ( ) blank
blank
uenluons
1oken: one or more
characLers separaLed
lnLo elds
Word: a Loken wlLh
no unquoLed
meLacharacLers
CperaLor: a Loken
wlLh one or more
meLacharacLers
Loken
word operaLor
uenluons
name: a word wlLh
a..z,A..,0..9,_ only
ConLrol operaLor:
performs conLrol
funcuons
8edlrecuon operaLor:
redlrecLs le
descrlpLors
lllename: ldenues a
le
8eserved word: speclal
meanlng , llke for or
while
Loken
word operaLor
name
reserved word
conLrol operaLor

|| && & , ,, | |&
( ) newllne
redlrecuon
operaLor


& &
lename
uenluons
arameLer: sLores
values
varlable: name for
sLorlng values, musL
begln wlLh leuer
Speclal parameLer:
canL be modled
osluonal
parameLers: for
reLrlevlng argumenLs
Loken
word operaLor
name
reserved word
conLrol operaLor

|| && & , ,, | |&
( ) newllne
redlrecuon
operaLor


& &
lename
funcuon/allas
parameLer
varlable
speclal parameLer
posluonal parameLer
Speclal arameLers - osluonal
osluonal parameLers are argumenLs passed
Lo Lhe shell when lnvoked
uenoLed by $dlglL, 0
normally used wlLh scrlpLs and funcuons
$ cat x.sh
echo $1 $2 $3 $4 $5 $6 $7 $8 $9 ${10} ${11} ${12}
$ bash x.sh {A..Z}
A B C D E F G H I J K L
Shell arameLers - Speclal
$* expands Lo Lhe posluonal parameLers
$@ vlrLually Lhe same as $*
$# number of posluonal parameLers
$- currenL opuon ags when shell lnvoked
$$ process ld of Lhe shell
$! process ld of lasL execuLed background command
$0 name of Lhe shell or shell scrlpL
$_ nal argumenL of lasL execuLed foreground command
$? exlL sLaLus of lasL execuLed foreground command
Speclal parameLers have a slngle characLer
Cuesuons? CommenLs?
sLa[hellx.nlh.gov