UNIX Shell Scripting, Tools and Utililties

Overview
Much of the underlying strength of Unix derives from the large collection of text-based tools and utilities which were inherited from UNIX, in conjunction with a powerful command and scripting language called the shell. Taken together, this toolset provides a high-level and astonishingly productive scripting environment in which all manner of solutions can be developed by using tools in appropriate combination. These range from simple automation of routine administrative tasks to post-processing, filtering and report generation from system log files or other structured data sources.
This course shows you how to use this powerful tool-building philosophy by focusing on the shell as a programming language, and on many of the key utilities and text filters. Emphasis is placed on the implementation of real-world solutions.

Key Skills
After completing this course you will be able to:

• Increase your productivity by using Unix tools effectively
• Construct bespoke solutions using tools in combination
• Write shell scripts to automate administrative tasks
• Search, sort, process and reformat text using filters

Practical Work
In addition to numerous short exercises to re-inforce understanding of specific language features, the course includes several longer workshops that illustrate the use of tools in combination to solve realistic problems. These workshops include:

• Developing a menu-driven shell
• Data mining from web server log files
• Compiling reports on file system usage
• A bulk mailing application
• A classroom reservation system

Course Contents
Basic utilities

• Simple filters: cat, wc, head, tail, tr, uniq...
• Sorting and searching tools: find, sort, grep, wildcards
• System reporting tools: ls, du, df, ps, netstat, uname, date ...
• Stream editing with sed
• Building solutions by combining tools

Searching and matching text with regular expressions

• Regular Expression syntax
• Examples using grep
• Other contexts for using regular expressions
• Substitutions using tagged regular expressions in sed and vi

Creating Shell Scripts

• Passing arguments to a script
• Built-in variables and environment variables
• I/O with echo and read
• Redirecting input and output
• Arithmetic and string operations

Flow control in the shell

• Conditional execution with if ... else
• Testing exit status of a command
• Testing file properties
• Looping with for, while and until
• Multiway branching using case

Advanced shell features

• Defining functions and aliases
• Command substitution
• Structuring data with arrays
• Command sequencing and grouping

Using awk to process structured input

• How awk processes lines and fields
• Patterns and actions
• Variables, operators and functions
• Formatted output
• Looping and branching in awk

Putting it all together

• Using pipes and command substitution
• Choosing the right tool for the job
• Incremental development of solutions
• Some common idioms