Scripting on Linux

Origins of scripting languages Learning scripting languages A simple example of a shell script A script with a loop and debugging Processing a table of data by selecting rows and columns Doing arithmetic internally within the shell Perl, Python, Ruby etc. Scheduling scripted jobs using cron

Origins of scripting languages

Hardwired programs and embedded OS allowed little system control. Job control language JCL allowed IBM mainframe operators to "batch" a sequence of programs into a routine job, e.g. deducting payments made from accounts outstanding.

Benefits: * greater accuracy of job control * less typing * ability to restart job from initial data when system goes down part way through.
Mainframe features reinvented on smaller/cheaper machines.

Command languages developed

TSO/Clist (IBM on-line), PCL (Prime Primos), DCL (DEC VAX/VMS), Aegis (Apollo DomainOS). MS-DOS Batch File (.BAT) etc.

Microsoft froze .BAT language early 1990s. Early versions of MS Basic packaged with MS-DOS, later versions sold separately.
Unix opened development and transfer of ideas between shells. * Bourne Shell: /bin/sh * 'C' Shell: /bin/csh * Korn Shell: /bin/ksh * Ash Shell: /bin/ash * GNU Bash Shell: /bin/sh and /bin/bash Unix shell scripts have usual 3rdG features, including loop constructs, variables, arrays, branches and functions.

Learning scripting languages

Many features shared between shell languages and 'C', Perl, Python etc. The 80/20 rule: 80% of the benefit derives from knowledge of the 20% most useful subset. Programmers learn by: * Reading tutorials, books, reference material * Studying code examples * Running modified code examples * Many small experiments * Knowing how to program using a similar language type

Learning scripting languages 2

Those requiring an in-depth understanding of these languages will need to read the books and on-line tutorials and carry out a comprehensive series of programming exercises.
In other cases a usable subset of knowledge can be obtained by reading the source code of existing programs and executing these, and by conducting small experiments supplemented with tactical use of the reference information provided with languages. Programming knowledge can grow on an as-needed basis.

A simple shell script example

Return values and tests

The Unix shell convention is for a program which exits successfully to end with a return of 0 (which the shell considers true), and for an error exit to result in a return of 1 or more (which the shell considers false). This is the opposite way round to how this is done within 'C'. The number N returned by a program (e.g. using the bash exit N statement) is different from the standard output of the same program. The shell command: echo $?

outputs the return code of the last foreground command.

A script with a loop and debugging

menu script part 2

A table with rows and columns

Here are some records from the may.logins input file:

reboot shutdown usr11361 usr11187 usr11187 usr11187 usr11513 usr11451 usr11456 usr11138 usr12069 usr12069

~ ~ console console console console console console console console console console

Tue May 11 13:14 Tue May 11 13:15 Tue May 11 09:04 - 10:22 (01:18) Mon May 10 18:53 - 20:30 (01:36) Mon May 10 18:50 - 18:53 (00:02) Mon May 10 18:38 - 18:50 (00:12) Mon May 10 15:15 - 16:27 (01:11) Mon May 10 12:11 still logged in Mon May 10 10:53 - 15:14 (04:21) Mon May 10 09:03 - 10:40 (01:36) Sat May 8 11:05 - 09:01 (1+21:55) Sat May 8 11:00 - 11:04 (00:04)

This gives us time logged, one record per login session.

Login analysis script 2

This combines a number of features of previous examples, using awk, grep and sed filters to access specific rows then columns, and to exclude unwanted data from the analysis. The input data is a set of login records. This application was used to analyse average usage of 20 workstations during particular months.

Login analysis script 3

Doing arithmetic internally within the shell

Smaller or earlier Unix shells don't have builtin arithmetic operators, so they farm this job out to external programs such as expr as we saw above. This can be done using the Bash shell let builtin, as in the following example script:

Advanced scripting: Perl, Python, Ruby etc.

Shell scripts provide glue logic together with other utilities, pipelines and redirection.

Advantage: Very rapid development of systems/network administration and automated operations. Disadvantages: Slow, not for very large programs, non-portable. Having to load and execute external programs very many times ?
Adding 200 user login accounts to a system: * Shell: 1 hour to run (1992), helped us learn how to do it * Perl: ('93) 2 minutes to run, we already knew how to do it * 'C' ('94) 6 seconds but 10 times as long to write the program.

Advanced scripting: Perl, Python, Ruby 2

Fully portable languages: Perl, Python and Ruby used to handle simple scripting-type applications or more complex requirements, e.g. object-oriented webapp development. These languages build upon features in shell languages, using similar syntax for many purposes, e.g. handling regular expressions. These languages trade machine for programmer efficiency, compare against 'C', 'C++' . Can combine benefits by using 'C' modules in scripting interpreter or split application design after profiling to find where cycles used.

Advanced scripting: Perl, Python, Ruby 3

The login analysis program described above was rewritten in Perl.

Advanced scripting: Perl, Python, Ruby 4 This program runs much faster than the shell script, because everything is done inside the same process. There are many syntactic similarities, but Perl borrows array and loop notation from 'C' and some other notation from grep, awk and sed. As in Bash, $ is used to introduce single (scalar) variables and @ is used for arrays. $_ is used for a default scalar variable, so that the substitution operation e.g: s/\)//; which strips a closing round bracket from a string, doesn't state the string it modifies.

Scheduling cron jobs on Linux 1

If scripts are the key to systems automation, how do we cause these to be run at set times of the day, or on set days of the week or month ? The program we use for this is called cron.
Cron is a background process which runs other programs at set times. It is controlled using a file called crontab. Crontab contains entries to run all scripts placed in directories: /etc/cron.hourly /etc/cron.daily, /etc/cron.weekly etc.

Scheduling cron jobs on Linux 2

# /etc/crontab: system-wide crontab # Unlike any other crontab you don't have to run the `crontab' # command to install the new version when you edit this file # and files in /etc/cron.d. These files also have username fields, # that none of the other crontabs do.
SHELL=/bin/sh PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin # m h dom mon dow user command 17 * * * * root cd / && run-parts --report /etc/cron.hourly 25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) 47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly ) 52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly ) # local modifications # update spam black and whitelist DNS zones every 10 minutes 01,11,21,31,41,51 * * * * root /usr/local/bin/update_dnsfiles # create and email report of weekly spam rejections 22 10 * * 6 root /usr/local/bin/mrejects