Linux Shell Scripting Cookbook - Third Edition

Title Page

Linux Shell Scripting Cookbook

Third Edition

Over 110 incredibly effective recipes to solve real-world problems, automate tedious tasks, and take advantage of Linux's newest features

Clif Flynt

Sarath Lakshman

Shantanu Tushar

BIRMINGHAM - MUMBAI

Copyright

Linux Shell Scripting Cookbook

Third Edition

Copyright © 2017 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: January 2011 

Second edition: May 2013

Third edition: May 2017 

Production reference: 1250517

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham 

B3 2PB, UK.

ISBN 978-1-78588-198-5

www.packtpub.com

Credits

About the Authors

Clif Flynt has been programming computers since 1970, administering Linux/Unix systems since 1985, and writing since he was 9 years old.

He's active in the Tcl/Tk and Linux user communities. He speaks frequently at technical conferences and user groups.

He owns and runs Noumena Corporation, where he develops custom software and delivers training sessions. His applications have been used by organizations ranging from one man startups to the US Navy. These applications range from distributed simulation systems to tools to help fiction authors write better (Editomat). He has trained programmers on four continents.

When not working with computers, Clif plays guitar, writes fiction experiments with new technologies, and plays with his wife's cats.

He's the author of Tcl/Tk: A Developer's Guide by Morgan Kauffman, 2012, as well as several papers, and magazine articles. His poetry and fiction have been published in small journals, including Write to Meow by Grey Wolfe Press, 2015.

h t t p ://w w w . n o u c o r p . c o m

h t t p s ://w w w . l i n k e d i n . c o m /i n /c l i f f l y n t /

I'd like to thank my wife for putting up with me during my writing marathons, and my editors at Packt Publishing, Sanjeet Rao, Radhika Atitkar, and Nidhisha Shetty  for their support and assistance.

Sarath Lakshman is a 27 year old who was bitten by the Linux bug during his teenage years. He is a software engineer working in ZCloud engineering group at Zynga, India. He is a life hacker who loves to explore innovations. He is a GNU/Linux enthusiast and hactivist of free and open source software. He spends most of his time hacking with computers and having fun with his great friends. Sarath is well known as the developer of SLYNUX (2005)—a user friendly GNU/Linux distribution for Linux newbies. The free and open source software projects he has contributed to are PiTiVi Video editor, SLYNUX GNU/Linux distro, Swathantra Malayalam Computing, School-Admin, Istanbul, and the Pardus Project. He has authored many articles for the Linux For You magazine on various domains of FOSS technologies. He had made a contribution to several different open source projects during his multiple Google Summer of Code projects. Currently, he is exploring his passion about scalable distributed systems in his spare time. Sarath can be reached via his website h t t p ://w w w . s a r a t h l a k s h m a n . c o m.

I would like to thank my friends and family for the great support and encouragement they have given me for all my endeavors. I would like to thank my friends Anu Mahadevan and Neenu Jacob for the tireless enthusiasm and patience to read through the chapter developments and providing comments during development. I would also like to thank Mr. Atanu Datta for helping me come up with the chapter titles. I extend my gratitude to the team at Packt Publishing who helped me in making this book happen.

Shantanu Tushar is an advanced GNU/Linux user since his college days. He works as an application developer and contributes to the software in the KDE projects. Shantanu has been fascinated by computers since he was a child, and spent most of his high

school time writing C code to perform daily activities. Since he started using GNU/Linux, he has been using shell scripts to make the computer do all the hard work for him. He also takes time to visit students at various colleges to introduce them to the power of Free Software, including its various tools. Shantanu is a well-known contributor in the KDE community and works on Calligra, Gluon and the Plasma subprojects. He looks after maintaining Calligra Active – KDE's offie document viewer for tablets, Plasma Media Center, and the Gluon Player. One day, he believes, programming will be so easy that everybody will love to write programs for their computers. Shantanu can be reached by e-mail on shantanu@kde.org, shantanutushar on       Identi.ca/Twitter, or his website h t t p ://w w w . s h a n t a n u t u s h a r . c o m.

About the Reviewer

John Kennedy has been a UNIX and Linux system administrator since 1997. He started on Solaris and has worked exclusively with Linux since 2005. He started scripting in 1999, when he realized that scripts could do much of his work for him, using the a lazy sysadmin is a great sysadmin philosophy. John currently works for Daon, a biometric security company, as a DevOps engineer.

John started tech editing in 2001 and also worked on the first edition of this book.

John has been married to Michele since 1994 and has a daughter, Denise, and a son, Kieran.

First, I’d like to thank my family for all their support. I also thank my dogs for their patience while I worked away on this book. Thanks also to my employers, Daon, Inc, who are awesome to work for.

Finally, I would like to thank Judie Jose for her patience on those occasions when life got in the way of my editing. Your exceptional support was greatly appreciated.

www.PacktPub.com

For support files and downloads related to your book, please visit www.PacktPub.com.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at service@packtpub.com for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

h t t p s ://w w w . p a c k t p u b . c o m /m a p t

Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt books and video courses, as well as industry-leading tools to help you plan your personal development and advance your career.

Why subscribe?

Fully searchable across every book published by Packt

Copy and paste, print, and bookmark content

On demand and accessible via a web browser

Customer Feedback

Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial process. To help us improve, please leave us an honest review on this book's Amazon page at https://www.amazon.com/dp/1785881981.

If you'd like to join our team of regular reviewers, you can e-mail us at customerreviews@packtpub.com. We award our regular reviewers with free eBooks and videos in exchange for their valuable feedback. Help us be relentless in improving our products!

Table of Contents

Preface

What this book covers

What you need for this book

Who this book is for

Sections

Getting ready

How to do it…

How it works…

There's more…

See also

Conventions

Reader feedback

Customer support

Downloading the example code

Downloading the color images of this book

Errata

Piracy

Questions

Shell Something Out

Introduction

Displaying output in a terminal

Getting ready

How to do it...

How it works...

There's more...

Escaping newline in echo

Printing a colored output

Using variables and environment variables

Getting ready

How to do it...

There's more...

Finding the length of a string

Identifying the current shell

Checking for super user

Modifying the Bash prompt string (username@hostname:~$)

Function to prepend to environment variables

How to do it...

How it works...

Math with the shell

How to do it...

Playing with file descriptors and redirection

Getting ready

How to do it...

How it works...

There's more...

Redirection from a file to a command

Redirecting from a text block enclosed within a script

Custom file descriptors

Arrays and associative arrays

Getting ready

How to do it...

There's more...

Defining associative arrays

Listing of array indexes

Visiting aliases

How to do it...

There's more...

Escaping aliases

Listing aliases

Grabbing information about the terminal

Getting ready

How to do it...

Getting and setting dates and delays

Getting ready

How to do it...

How it works...

There's more...

Producing delays in a script

Debugging the script

How to do it...

How it works...

There's more...

Shebang hack

Functions and arguments

How to do it...

There's more...

The recursive function

Reading the return value (status) of a command

Passing arguments to commands

Sending output from one command to another

Getting ready

How to do it...

There's more...

Spawning a separate process with subshell

Subshell quoting to preserve spacing and the newline character

Reading n characters without pressing the return key

How to do it...

Running a command until it succeeds

How to do it...

How it works...

There's more...

A faster approach

Adding a delay

Field separators and iterators

Getting ready

How to do it...

Comparisons and tests

How to do it...

Customizing bash with configuration files

How to do it...

Have a Good Command

Introduction

Concatenating with cat

How to do it...

There's more...

Getting rid of extra blank lines

Displaying tabs as ^I

Line numbers

Recording and playing back terminal sessions

Getting ready

How to do it...

How it works...

Finding files and file listing

Getting ready

How to do it...

There's more...

Search based on name or regular expression match

Negating arguments

Searching based on the directory depth

Searching based on file type

Searching by file timestamp

Searching based on file size

Matching based on file permissions and ownership

Performing actions on files with find

Deleting based on file matches

Executing a command

Skipping specified directories when using the find command

Playing with xargs

Getting ready

How to do it...

How it works...

There's more...

Passing formatted arguments to a command by reading stdin

Using xargs with find

Counting the number of lines of C code in a source code directory

While and subshell trick with stdin

Translating with tr

Getting ready

How to do it...

How it works...

There's more...

Deleting characters using tr

Complementing character sets

Squeezing characters with tr

Character classes

Checksum and verification

Getting ready

How to do it...

How it works...

There's more...

Checksum for directories

Cryptographic tools and hashes

How to do it...

Sorting unique and duplicate lines

Getting ready

How to do it...

How it works...

There's more...

Sorting according to keys or columns

uniq

Temporary file naming and random numbers

How to do it...

How it works...

Splitting files and data

How to do it...

There's more...

Specifying a filename prefix for the split files

Slicing filenames based on extensions

How to do it...

How it works...

Renaming and moving files in bulk

Getting ready

How to do it...

How it works...

Spell–checking and dictionary manipulation

How to do it...

How it works...

Automating interactive input

Getting ready

How to do it...

How it works...

There's more...

Automating with expect

Making commands quicker by running parallel processes

How to do it...

How it works...

There's more...

Examining a directory, files and subdirectories in it

Getting ready

How to do it...

Generating a tree view of a directory.

Generating a summary of files and sub-directories

File In, File Out

Introduction

Generating files of any size

How to do it...

The intersection and set difference (A-B) on text files

Getting ready

How to do it...

How it works...

Finding and deleting duplicate files

Getting ready

How to do it...

How it works...

Working with file permissions, ownership, and the sticky bit

How to do it...

There's more...

Changing ownership

Setting the sticky bit

Applying permissions recursively to files

Applying ownership recursively

Running an executable as a different user (setuid)

Making files immutable

Getting ready

How to do it...

Generating blank files in bulk

Getting ready

How to do it...

Finding symbolic links and their targets

How to do it...

How it works...

Enumerating file type statistics

Getting ready

How to do it...

How it works...

Using loopback files

How to do it...

How it works...

There's more...

Creating partitions inside loopback images

Mounting loopback disk images with partitions more quickly

Mounting ISO files as loopback

Flush changing immediately with sync

Creating ISO files and hybrid ISO

Getting ready

How to do it...

There's more...

Hybrid ISO that boots off a flash drive or hard disk

Burning an ISO from the command line

Playing with the CD-ROM tray

Finding the difference between files, and patching

How to do it...

There's more...

Generating difference against directories

Using head and tail for printing the last or first 10 lines

How to do it...

Listing only directories - alternative methods

Getting ready

How to do it...

How it works...

Fast command-line navigation using pushd and popd

Getting ready

How to do it...

There's more...

pushd and popd are useful when there are more than three directory paths used. However, when you use only two locations, there is an alternative and easier way, that is, cd -.

Counting the number of lines, words, and characters in a file

How to do it...

Printing the directory tree

Getting ready

How to do it...

There's more...

HTML output for tree

Manipulating video and image files

Getting ready

Extracting Audio from a movie file (mp4)

How to do it...

Making a video from a set of still images

How to do it...

How it works...

Creating a panned video from a still camera shot

How to do it...

How it works...

Texting and Driving

Introduction

Using regular expressions

How to do it...

Position markers

Identifiers

Count modifiers

Other

There's more...

How it works...

There's more...

Treatment of special characters

Visualizing regular expressions

Searching and mining text inside a file with grep

How to do it...

There's more...

Recursively searching many files

Ignoring case in patterns

grep by matching multiple patterns

Including and excluding files in a grep search

Using grep with xargs with the zero-byte suffix

Silent output for grep

Printing lines before and after text matches

Cutting a file column-wise with cut

How to do it...

There's more

Specifying the range of characters or bytes as fields

Using sed to perform text replacement

How to do it...

There's more...

Removing blank lines

Performing replacement directly in the file

Matched string notation ()

Substring match notation (\1)

Combining multiple expressions

Quoting

Using awk for advanced text processing

Getting ready

How to do it...

How it works...

There's more...

Special variables

Passing an external variable to awk

Reading a line explicitly using getline

Filtering lines processed by awk with filter patterns

Setting delimiters for fields

Reading the command output from awk

Associative arrays in Awk

Using loop inside awk

String manipulation functions in awk

Finding the frequency of words used in a given file

Getting ready

How to do it...

How it works...

See also

Compressing or decompressing JavaScript

Getting ready

How to do it...

How it works...

See also

Merging multiple files as columns

How to do it...

See also

Printing the nth word or column in a file or line

How to do it...

See also

Printing text between line numbers or patterns

Getting ready

How to do it...

See also

Printing lines in the reverse order

Getting ready

How to do it...

How it works...

Parsing e-mail address and URLs from text

How to do it...

How it works...

See also

Removing a sentence in a file containing a word

Getting ready

How to do it...

How it works...

See also

Replacing a pattern with text in all the files in a directory

How to do it...

How it works...

There's more...

Text slicing and parameter operations

How to do it...

See also

Tangled Web? Not At All!

Introduction

Downloading from a web page

Getting ready

How to do it...

How it works...

There's more...

Restricting the download speed

Resume downloading and continue

Copying a complete website (mirroring)

Accessing pages with HTTP or FTP authentication

Downloading a web page as plain text

Getting ready

How to do it...

A primer on cURL

Getting ready

How to do it...

How it works...

There's more...

Continuing and resuming downloads

Setting the referer string with cURL

Cookies with cURL

Setting a user agent string with cURL

Specifying a bandwidth limit on cURL

Specifying the maximum download size

Authenticating with cURL

Printing response headers excluding data

See also

Accessing unread Gmail e-mails from the command line

How to do it...

How it works...

See also

Parsing data from a website

How to do it...

How it works...

See also

Image crawler and downloader

How to do it...

How it works...

See also

Web photo album generator

Getting ready

How to do it...

How it works...

See also

Twitter command-line client

Getting ready

How to do it...

How it works...

See also

Accessing word definitions via a web server

Getting ready

How to do it...

How it works...

See also

Finding broken links in a website

Getting ready

How to do it...

How it works...

See also

Tracking changes to a website

Getting ready

How to do it...

How it works...

See also

Posting to a web page and reading the response

Getting ready

How to do it...

See also

Downloading a video from the Internet

Getting ready

How to do it...

How it works...

Summarizing text with OTS

Getting ready

How to do it...

How it works...

Translating text from the command line

Getting ready

How to do it...

How it works...

Repository Management

Introduction

Creating a new git repository

Getting ready

How to do it...

How it works...

Cloning a remote git repository

How to do it...

Adding and committing changes with git

How to do it...

Creating and merging branches with git

Getting ready...

How to do it...

How it works...

There's more...

Merging branches

How to do it...

How it works...

There's more...

Sharing your work

How to do it...

Pushing a branch to a server

How to do it...

Checking the status of a git repository

How to do it...

How it works...

Viewing git history

How to do it...

Finding bugs

How to do it...

There's more...

How to do it...

How it works...

Tagging snapshots

How to do it...

Committing message ethics

How to do it...

Using fossil

Getting ready

How to do it...

Creating a new fossil repository

How to do it...

How it works...

There's more...

Web interface to fossil

How to do it...

Making a repository available to remote users

Cloning a remote fossil repository

How to do it...

How it works...

Opening a fossil project

How to do it...

How it works...

There's more...

Adding and committing changes with fossil

How to do it...

There's more...

Using branches and forks with fossil

How to do it

How it works...

There's more...

Merging forks and branches

How to do it...

Sharing your work with fossil

How to do it...

How it works...

Updating your local fossil repository

How to do it...

Checking the status of a fossil repository

How to do it...

Viewing fossil history

How to do it...

Finding bugs

How to do it...

There's more...

Tagging snapshots

How to do it...

There's more...

The Backup Plan

Introduction

Archiving with tar

Getting ready

How to do it...

How it works...

There's more...

Appending files to an archive

Extracting files and folders from an archive

stdin and stdout with tar

Concatenating two archives

Updating files in an archive with a timestamp check

Comparing files in the archive and filesystem

Deleting files from the archive

Compression with the tar archive

Excluding a set of files from archiving

Excluding version control directories

Printing the total bytes

See also

Archiving with cpio

How to do it...

How it works...

Compressing data with gzip

How to do it...

There's more...

Gzip with tarball

zcat - reading gzipped files without extracting

Compression ratio

Using bzip2

Using lzma

See also

Archiving and compressing with zip

How to do it...

How it works...

Faster archiving with pbzip2

Getting ready

How to do it...

How it works...

There's more...

Manually specifying the number of CPUs

Specifying the compression ratio

Creating filesystems with compression

Getting ready

How to do it...

There's more...

Excluding files while creating a squashfs file

Backing up snapshots with rsync

How to do it...

How it works...

There's more...

Excluding files while archiving with rsync

Deleting non-existent files while updating rsync backup

Scheduling backups at intervals

Differential archives

How to do it...

How it works...

Creating entire disk images using fsarchiver

Getting ready

How to do it...

How it works...

The Old-Boy Network

Introduction

Setting up the network

Getting ready

How to do it...

There's more...

Printing the list of network interfaces

Displaying IP addresses

Spoofing the hardware address (MAC address)

Name server and DNS (Domain Name Service)

DNS lookup

Showing routing table information

See also

Let us ping!

How to do it...

There's more...

Round Trip Time

Sequence number

Time to live

Limiting the number of packets to be sent

Return status of the ping command

Tracing IP routes

How to do it...

Listing all available machines on a network

Getting ready

How to do it...

How it works...

There's more...

Parallel pings

Using fping

See also

Running commands on a remote host with SSH

Getting ready

How to do it...

There's more...

SSH with compression

Redirecting data into stdin of remote host shell commands

Running graphical commands on a remote machine

How to do it...

See also

Transferring files through the network

Getting ready

How to do it...

There's more...

Automated FTP transfer

SFTP (Secure FTP)

The rsync command

SCP (secure copy program)

Recursive copying with scp

See also

Connecting to a wireless network

Getting ready

How to do it...

How it works...

See also

Password-less auto-login with SSH

Getting ready

How to do it...

Port forwarding using SSH

How to do it...

There's more...

Non-interactive port forward

Reverse port forwarding

Mounting a remote drive at a local mount point

Getting ready

How to do it...

See also

Network traffic and port analysis

Getting ready

How to do it...

How it works...

There's more...

Opened port and services using netstat

Measuring network bandwidth

How to do it...

Creating arbitrary sockets

Getting ready

How to do it...

There's more...

Quickly copying files over the network

Creating a broadcasting server

How it works...

Building a bridge

Getting ready

How to do it...

Sharing an Internet connection

Getting ready

How to do it...

How it works

Basic firewall using iptables

How to do it...

How it works...

There's more...

Creating a Virtual Private Network

Getting ready

How to do it...

Creating certificates

Configuring OpenVPN on the server

Configuring OpenVPN on the client

Starting the server

Starting and testing a client

Put On the Monitors Cap

Introduction

Monitoring disk usage

Getting ready

How to do it...

There's more...

Displaying disk usage in KB, MB, or blocks

Displaying the grand total sum of disk usage

Printing sizes in specified units

Excluding files from the disk usage calculation

Finding the ten largest size files from a given directory

Disk free information

Calculating the execution time for a command

How to do it...

How it works...

Collecting information about logged in users, boot logs, and boot failures

Getting ready

How to do it...

Listing the top ten CPU– consuming processes in an hour

Getting ready

How to do it...

How it works...

See also

Monitoring command outputs with watch

How to do it...

There's more

Highlighting the differences in the watch output

Logging access to files and directories

Getting ready

How to do it...

How it works...

Logging with syslog

Getting ready

How to do it...

See also

Managing log files with logrotate

Getting ready

How to do it...

How it works...

Monitoring user logins to find intruders

Getting ready

How to do it...

How it works...

Monitoring remote disk usage health

Getting ready

How to do it...

How it works...

See also

Determining active user hours on a system

Getting ready

How to do it...

How it works...

Measuring and optimizing power usage

Getting ready

How to do it...

Monitoring disk activity

Getting ready

How to do it...

Checking disks and filesystems for errors

Getting ready

How to do it...

How it works...

Examining disk health

Getting ready

How to do it...

How it works

Getting disk statistics

Getting ready

How to do it...

How it works

There's more

Administration Calls

Introduction

Gathering information about processes

Getting ready

How to do it...

How it works...

There's more...

Showing environment variables for a process

Creating a tree view of processes

Sorting ps output

Filters with ps for real user or ID, effective user or ID

TTY filter for ps

Information about process threads

Specifying the output width and columns to be displayed

What's what – which, whereis, whatis, and file

How to do it...

Finding the process ID from the given command names

Determining how busy a system is

The top command

See also...

Killing processes, and sending and responding to signals

Getting ready

How to do it...

There's more...

The kill family of commands

Capturing and responding to signals

Sending messages to user terminals

Getting ready

How to do it...

Sending one message to one user

Holding a conversation with another user

Sending a message to all users

The /proc filesystem

How to do it...

Gathering system information

How to do it...

Scheduling with a cron

Getting ready

How to do it...

How it works...

There's more...

Specifying environment variables

Running commands at system start-up/boot

Viewing the cron table

Removing the cron table

Database styles and uses

Getting ready

How to do it...

There's more...

Creating a table

Inserting a row into an SQL database

Selecting rows from a SQL database

Writing and reading SQLite databases

Getting ready

How to do it...

How it works...

There's more...

Writing and reading a MySQL database from Bash

Getting ready

How to do it...

How it works...

User administration scripts

How to do it...

How it works...