3/11/13

Introduction to Unix
Files and Shells

COMMONWEALTH OF AUSTRALIA Copyright Regulations 1969 WARNING

Bob Kummerfeld COMP2129

This material has been reproduced and communicated to you by or on behalf of the University of Sydney pursuant to Part VB of the Copyright Act 1968 (the Act). The material in this communication may be subject to copyright under the Act. Any further copying or communication of this material by you may be the subject of copyright protection under the Act.

FACULTY OF ENGINEERING & INFORMATION TECHNOLOGIES

Do not remove this notice.

The file system

root usr pub ascii Ass1 Erat bin passwd tmp home ken hrmt pub ascii usr bin tmp

/ root

Paths and full path names

dev

etc

dev

etc passwd Ass1 ken

home hrmt

Erat

Erat.c doit

a.out

Erat.c. doit

a.out

/ root usr pub ascii /usr/pub/ascii Full path name bin tmp

Paths and full path names

/ root usr pub ascii bin tmp

Paths and full path names

dev

etc passwd Ass1 ken

home hrmt

dev

etc passwd ken

home .. hrmt

Erat

/usr/pub/ascii Full path name a.out

Ass1

Erat

Erat.c. doit

Erat c. doit

a.out

Relative path name

3/11/13

/ root usr pub ascii /usr/pub/ascii Full path name bin tmp

Paths and full path names

Several types of files
- Ordinary files hrmt /home/hrmt Ass1 Erat . ~hrmt - Contain data, eg text, program - Directories - Actually a special file that contains a list of files - Devices - The file is a name that refers to a device such as a disk drive or network interface - special files - The content of these files is generated when the file is read - Eg the files in /proc on a linux system

Files

dev

etc passwd ken

home ..

Erat c. doit

a.out

Home directory ~

Relative path name

File command

File command and C files

The file command tries to work out what type of file you have: bash$ file Sunix Sunix directory bash$ cd Sunix bash$ file notes notes: English text

bash$ file fperror.c fperror.c: C program text bash$ gcc fperror.c bash$ file a.out
a.out: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped

bash$ cp fperror.c fperror bash$ gcc fperror

fperror: file not recognized: File format not recognized collect2: ld returned 1 exit status

At login

Access and permissions

Shell reads from

/etc/profile

bash$ ls -l
drwxr-x--drwxrwxr-x drwxrwxr-x drwxr-x---rw-r----drwxrwxr-x drwxr-x--6 judy 12 judy 6 judy 4 judy 1 judy 9 judy 7 judy judy judy judy judy judy judy judy 4096 Dec 14 4096 Jul 10 4096 Feb 11 4882 Feb 12 2012 Cs12000 2012 Cs12001 1999 IF 2001 Kay.ps

Then it reads your own

.bash_profile .profile

4096 Mar 15 12:29 EPS

4096 Mar 13 22:01 SDM 4096 Dec 17 11:06 Uidp

Is your environment different? How can you see these files, starting with .

Directory/file etc user, group, other

3/11/13

Meanings of permissions

Meanings of permissions

Read
- File: - Directory

Read
- File: program can read it - Directory: can ls it

Write
- File: - Directory

Write
- File: program can alter it - Directory: can add/remove files in it

Execute
- File: - Directory

Execute
- File: can run the program - Directory: can read files in it, if name given

Some important cases

What the shell does

Your home directory

ls ld .

Protecting files from yourself

chmod u-w precious_file.c

Making files executable

chmod u+x doit General form of chmod symbolic commands

chmod [u g o] [+ - ] [r w x] filenames

gcc -ansi -wall -pedantic *.c first word is the command name - gcc spaces are delimiters expansions of metacharacters * look for this along $PATH execute program with that name pass rest of line as parameters programmer of command (gcc) decides what is allowed

Paths

Initial path

default search paths - $PATH echo $PATH colon separated list of paths

PATH= /local/usr/bin: /gnu/usr/bin: /usr/bin: /bin: /usr/ccs/bin: /usr/ucb: /usr/openwin/bin: .:

3/11/13

Paths searched by shell

root usr bin pub ascii bin passwd /local/usr/bin /gnu/usr/bin /usr/bin /bin /usr/ccs/bin /usr/ucb /usr/openwin/bin . Ass1 tmp home ken html

Special characters

Selected shell short cuts

dev

etc

* [ ] 0-9 a-z
Examples

Erat

Erat.c doit

a.out

ls *.c gcc *.c ls -ld [0-9]* ls [a-z]*.c rm i *

Redirection and pipes

Redirection

0 stdin a.out

1 stdout

stdin data1

a.out

stdout res1 stderr errors

stderr 2 bash$ a.out bash$ a.out

bash$ a.out < data1 >res1 2>errors

Redirection - appending to a file

Redirection and

pipes
stdout

stdin data1

a.out

stdout res1 stderr errors stdin

a.out

stdout stdin

more

bash$ a.out bash$ a.out < data1 res1 2>errors

a.out | more
pipe symbol

3/11/13

Initial shell on lab machines

bash$ env LOGNAME=demo
LD_LIBRARY_PATH=/gnu/usr/lib:/usr/openwin/lib:/local/usr/lib:/usr/local/openssl/lib

Shell command to save typing

WWW_HOME=http://www.ug.cs.usyd.edu.au/

TERM=vt100

HOSTTYPE=sparc

PATH=/local/usr/bin:/gnu/usr/bin:/usr/bin:/bin:/usr/ccs/bin:/usr/ucb:/usr/openwin/bin:.:/usr/bags/stubin

HOME=/usr/cs2/demo SHELL=/gnu/usr/bin/bash
PS1=bash\$

HZ=100

http_proxy=http://www-cache.cs.usyd.edu.au:8000/

ftp_proxy=http://www-cache.cs.usyd.edu.au:8000/

MANPATH=/local/usr/man:/gnu/usr/man:/usr/share/man:/usr/openwin/man

gopher_proxy=http://www-cache.cs.usyd.edu.au:8000/

OSTYPE=SunOS5

NNTPSERVER=news.cs.usyd.edu.au

OPENWINHOME=/usr/openwin

SHLVL=1

EDITOR=/local/usr/bin/red
TZ=Australia/NSW WWW_http_GATEWAY=http://www-cache.cs.usyd.edu.au:8000/

_=/usr/bin/env

Create file called doit, containing #!/bin/bash! echo "Compile the masterpiece"! gcc -W -Wall -pedantic -ansi *.c! Make executable and run it bash$ chmod u+x doit! bash$ ./doit! Compile the masterpiece! bash$ !

Your own commands

Save typing on common command sequences

Create a command that will show all the details of the 5 most recently changed files in the current directory
ls ltr | tail 5

Create a file with commands to

- compile erat.c - run it, taking data from 01_data, saving the results in 01_res - ditto 02_data and 02_res - ditto 03_data and 03_res - then compare actual and expected results in (01_expected, 02_expected, 03_expected)

Need execute permission Need it on path

doit

doit

gcc -W -Wall -pedantic -ansi erat.c a.out < 01_data > 01_res a.out < 02_data > 02_res a.out < 03_data > 03_res echo comparing first test results .. diff 01_res 01_expected echo comparing second test results .. diff 02_res 02_expected echo comparing third test results .. diff 03_res 03_expected

Then make doit executable If . not in $PATH, need ./doit, else doit What about 50 tests?

3/11/13

doit again

doit so far

gcc -W -Wall -pedantic -ansi erat.c echo Test on 01_data - normal > result a.out < 01_data >> result printf Test on 02_data - normal\n >> result a.out < 02_data >> result echo Test on 03_data - boundary >> result a.out < 03_data >> result

Saves typing Means you will be more likely to test systematically you know there has to be a better way to handle this, especially with 50..100.. tests

Another tiny task

A file has a list of people's last names several per line. We want to know how many people with different names are in the file. bash$ cat name_list Wang Ng Smith Lee Ng Howard Lee bash$

bash$ fmt -1 name_list > names_split bash$ sort -u names_split > names_sort bash$

bash$ fmt -1 name_list > names_split bash$ cat names_split Wang Ng Smith Lee Ng Howard Lee bash$ wc names_split
7 7 32 names_split

bash$ fmt -1 name_list > names_split bash$ sort -u names_split > names_sort bash$ wc names_sort

bash$ sort -u names_split > names_sort bash$ wc names_sort

5
Howard Lee Ng Smith

25 names_sort

bash$ cat names_sort

the programmer might check as they go.

Wang

3/11/13

How do you get to know about fmt and all the other commands?

Summary

Online manual

Shell
- scripts a scripting language - Speed of creating solutions - Interpreted, slower - default search paths - $PATH - colon separated list of paths

Practice to automate your knowledge

File System
- Files - Directories - Paths

xkcd.org comment on kilobyte

Shell commands and scripts

The Unix Shell

a shell is unix command interpreter the shell reads commands typed in a command window or from a file, interprets them and runs the appropriate programs simple shell commands such as: gcc myprog.c will run the program (gcc) you specified

more complex commands may involve running several programs and connecting one to the next through pipes the shell also as simple control structures that dont involve running a program but instead control how other programs are run eg: if and for statement

3/11/13

The Unix Shell: key concepts

I/O redirection
when programs are run they have three standard files opened:
- standard input (0) - standard output (1) - standard error (2)

I/O redirection and pipes scripts variables control structures these shell features are supported by many simple commands that can be combined in scripts to provide powerful services

these are normally all connected to the terminal window can be redirected using > or < or |
!gcc myprog.c >output!

Pipes
pipes let you connect the standard output of one program to the standard input of another program myprog | hisprog! this has the same effect as myprog >tempfile! ! !hisprog <tempfile! commands can be placed in a file and the shell will read and interpret them this allows you to create new commands easily: #!/bin/sh! !gcc myprog.c -o myprog! !echo Finished!!
which command interpreter to use

script file should be readable and executable

Shell Variables
you set a shell variable with a statement like: VARNAME=value you use a shell variable in a statement like: $VARNAME

Shell Variables

the shell has variables that you can give a string value to there are a number of predefined variables such as PATH and HOME you can create your own variables in a script or on the command line the variables are stored in the environment of a program, on the stack along with program arguments when the program starts

the shell will replace the $VARNAME with the string value

3/11/13

Script Arguments

Control Structures

shell script arguments are available in special variables $1, $2 etc for example, the script called compile: #!/bin/sh! ! !gcc $1.c -o $1! you invoke this like:
compile myprog!

there are many different shell command interpreters available

-we use bash

Most shells offer the control stuctures:

- if - for - case - while

a list of all script arguments is available in the shell variable $*!

these are implemented by the shell program itself, not by separate commands

if statement

if statement

The if statement is used to test a condition and control flow of execution in a script if command! !then! ! !command2! ! !command3! !fi!

example:

then clause executed if myprog exits with result 0

if myprog then echo myprog ok else echo myprog failed fi else clause executed if myprog
with non-zero result

exits

test command

for statement

there is a command called test that will test some condition and exit with true or false (0 or 1) exit status if test $1 == fred ! ! !then! ! ! !echo first arg is fred! ! !fi! test can test many conditions, especially relating to files - see the manual entry (man test) calling your own program test is a bad idea!

the for statement is used to iterate over a list, usually a list of files: doit script: for name in $*! ! !do! ! ! !gcc $name.c -o $name! ! !done! this will compile each file name given when the script is invoked, eg doit myprog hisprog

3/11/13

while

case

the while statement executes a block of a shell script while some condition is true, eg: while test $result != 10! !do! ! !! !done!

case is a selection statement (like switch in C) case $type in! ! !1) !myprog;;! ! !2) !hisprog;;! ! !pattern) otherprog;;! ! !esac!

Useful Commands for handling text files

sorts lines of text in a file, very flexible choice of key field to sort on, can remove duplicates, sends sorted lines to output cuts fields out lines of text from a file and sends the result to output transliterates or removes characters from a file eg remove \r from a file compares files and prints lines that appear in only one or both the files

Example

sort!

cut!

tr!

comm!

#!/bin/sh tr A-Z a-z < $1 | tr -cs a-z \n | sort -u | comm -13 /usr/dict/words

see the manual entries for details

Example

Example

convert upper case to lower case Read from first argument to script

#!/bin/sh tr A-Z a-z < $1 | tr -cs a-z \n | sort -u | comm -13 /usr/dict/words

#!/bin/sh tr A-Z a-z < $1 | tr -cs a-z \n | sort -u | comm -13 /usr/dict/words

10

3/11/13

Example

Example

convert upper case to lower case

convert upper case to lower case

convert non-alphabetic to a newline #!/bin/sh tr A-Z a-z < $1 | tr -cs a-z \n | sort -u | comm -13 /usr/dict/words

convert non-alphabetic to a newline #!/bin/sh tr A-Z a-z < $1 | sort and remove duplicates tr -cs a-z \n | sort -u | comm -13 /usr/dict/words

Example

Example

convert upper case to lower case

What does it do?!

#!/bin/sh tr A-Z a-z < $1 | tr -cs a-z \n | sort -u | comm -13 /usr/dict/words

convert non-alphabetic to a newline #!/bin/sh tr A-Z a-z < $1 | sort and remove duplicates tr -cs a-z \n | print lines that appear in input but not sort -u | in /usr/dict/words comm -13 /usr/dict/words

Shell scripts instead of programs

More useful shell features

changing your prompt The combination of the shell scripting language (variables, pipes, control structures etc) and many general purpose text processing commands can often be a more effective way of solving a problem than writing a program in a conventional language (Java, C etc)
the prompt is printed by the shell to indicate that it is waiting for you to type a command you can change the prompt to contain useful information such as current machine or directory. This is done using the PS1 shell variable bash2.5$ PS1=\h \W congo1 bob%

11

3/11/13

subshells
sometimes commands are run in another copy of the shell a subshell command substitution
an argument enclosed in backquotes indicates that a command is to run and the output used as the actual argument(s) eg myprog `cat argfile` will run the cat command, take the output (ie the contents of argfile) and use it as the arguments for myprog

the environment is copied from the parent shell, the subshell may change it but it reverts to the original when the subshell exits tar cf MyDir | (cd Somewhere; tar xf -)

subshells
sometimes commands are run in another copy of the shell a subshell the environment is copied from the parent shell, the subshell may change it but it reverts to the original when the subshell exits tar cf MyDir | (cd Somewhere; tar xf -) also useful to collect output from several commands: (echo My Data; cat file1) >output! this also works with for and while for X in $*! do!
done in a subshell

!echo $X! !cat $X! done > newfile!

take the collected output from the for loop and redirect to newfile

arithmetic
the expr command in standard shell will evaluate its arguments as an expression myprog `expr $count + 1`! this will run expr to calculate the value of the shell variable count, add 1 to it and send the answer to the output where it will be used as the argument to myprog

the test command will evaluate boolean expressions. It will also let you test file characteristics
if test d name! ! ! ! !then! ! !fi! -d name is a directory -s file exists and is non-zero size -r file exists and is readable -w file exists and is writable etc !echo name is a directory!

in the Bash shell you can use the command let for assignment statements of variables
! !let count = count +1!

12

3/11/13

input from within a script

reading text from within a shell script
read x! will read a line from standard input and store it in x grep $1 <<END! Alice! Bob! Carrol! END!

also called a here document

read name! if [ $name == Fred ]! then! !echo name was Fred! fi!

This word can be anything, it is used to terminate the document

grep
the grep command will scan a text file, matching each line against a specified regular expression and print those that match
Regular expression ! ! !grep [Bb]ob data*! grep -i comp2129 access_log! will scan each file whose name starts with data and print all lines that contain either Bob or bob this will scan the web log for lines containing comp2129 eg:

grep

grep can be used to scan output for lines of interest

207.96.187.226 - - [17/May/2008:03:40:01 +1000] "GET /~comp2129/PDF/2008_wk5.2_scope_functions.pdf HTTP/1.1" 200 31350

grep
Ignore case

Finding files

find command
find . name *.mp3 print! this will start at the current directory (.) and search recursively through your directories looking for files that end in .mp3 and print the name

grep can be used to scan output for lines of interest

grep -i comp2129 access_log! this will scan the web log for lines containing comp2129 eg:

locate command will print the full path names of files with names that match a pattern du will print out disk usage starting at the current directory. du a will list all files, starting at the current directory could then use grep to find a file.

207.96.187.226 - - [17/May/2008:03:40:01 +1000] "GET /~comp2129/PDF/2008_wk5.2_scope_functions.pdf HTTP/1.1" 200 31350

13

3/11/13

Comparing files
diff will show the differences between files sdiff is a side by side diff that shows both texts next to each other with the differences indicated.

top
the top command will show running processes dynamically the display is updated every second or so to show the details of processes very flexible options, such as sorting on various process attributes

Strange file names

if you inadvertently (or on purpose) end up with a file with a strange name it can be hard to work with it
eg a file called -x will often be confused with an argument -x (x can be almost any char) so nano R will not try and open a file called -R. To make this work use: nano ./-R

Common problems

forgetting to set execute permission forgetting that the current directory is not in $PATH using test as the name of a prog or script

Example: generating web pages

web pages can be static files that contain a description of the page written in the HTML markup language web pages are often generated by a program when you request the page web page generation is carried out by programs that take current information from many sources and write out the HTML for the page

End of Section

14

3/11/13

How the Web works (the 2 minute version)

Web Pages

a user types a URL into a web browser program the browser parses the URL and determines the machine to contact for the page the browser makes a network connection to a web server running on the remote machine and requests the web page specified by the URL the web server copies the page from a file to the browser the browser displays the page

web pages are usually written in HTML (HyperText Markup Language) eg: <H1>Heading</H1> This is <B>bold</B> text. but can be images, sounds, plain text etc

Creating your own web pages

Creating your own web pages

web pages are stored in files in (SIT servers):

$HOME/lib/html

setting permissions:
chmod 711 $HOME chmod 711 $HOME/lib chmod 711 $HOME/lib/html chmod 744 $HOME/lib/html/myfile.txt

a file ending in .html is written in HTML a file ending in .txt will be displayed as plain text to allow the web server to access your files they need to be world readable along with the directories

CGI Scripts

Making a web page with a CGI script

generated web pages are often created by cgi scripts that run on the web server machine in response to a browser asking for a URL cgi scripts are executable programs or scripts in files ending in .cgi could be written in C, java, perl, python
Web Browser

url web page Web Server

or could be written in shell!

CGI script

15

3/11/13

CGI shell script

tells the browser the type of web page

CGI script

#!/bin/sh echo "Content-type: text/plain" the blank line is essential echo echo Testing my CGI script

#!/bin/sh echo "Content-type: text/plain" echo cat example.txt

page content can come from anywhere

content of the page

Count the visitors

Whos logged on?

#!/bin/sh at that point echo "Content-type: text/plain" echo cat example.txt echo "Hello, today is " `date` count=`cat counter` count=`expr $count + 1` echo $count >counter echo "There have been " $count " visits to this page"

use the output of the command

echo "There are " who | cut -c1-10|sort -u|wc -l echo "people logged on" Output of the group of echo "Friends logged in:" commands within the for loop is sent through the pipe for name in `cat friends` do who | grep $name done | cut -c1-10 |sort -u

Summary

Entire web sites are generated from data held in databases Languages such as Perl, PHP or Python are commonly used COMP2129 class2go web site is generated by python scripts Shell can be used for simple web page generation

16