Bash Scripting

From Leo's Notes
Last edited on 16 September 2022, at 23:53.

Useful functions

Here are some common functions and commands I use in my scripts.

Function Code
Use this to ensure the script runs relative to its location.

This is useful if your script uses relative paths. 

#!/bin/bash
cd "$(dirname "$0")"
Find the maximum of two numbers. You can implement min by flipping the operands. 
max (){
        echo $(($1>$2 ? $1 : $2))
}
Similar to the die command in PHP which prints out a message before stopping execution.

Note that this will not work inside a subshell as it will just quit the subshell rather than the script. To get around this, you may need to kill the parent PID instead. 

function die {
	echo $@
	exit
}
Indents text piped to the function. 
# Supports up to 10 tabs
function Indent (){
	Tabs="\t\t\t\t\t\t\t\t\t\t"
	sed "s/^/${Tabs:0:$((2*$1))}/g"
}

echo "test" | Indent 1
	test
echo "test" | Indent 2
		test
Extracting the file name and file extension from a path 
Path="/etc/resolv.conf"

# Get path parts
Directory=$(dirname $Path)  # /etc
Filename=$(basename $Path)  # resolv.conf

# Get filename parts
Name="${Filename%.*}"  # resolv
Ext="${Filename##*.}"  # conf
Skip the first N lines from an output 
## -n +N outputs starting on line N.
## Eg. To skip the first 2 lines, we start on line 3:
# tail -n+3
Get the current Unix timestamp 
$ date +"%s"
1596137957
Unix timestamp to human readable time.

You can use either awk with strftime, or with date. 

$ echo 1355514777 | awk '{ print strftime("%c", $1) }'
Fri 14 Dec 2012 12:52:57 PM MST

$ date +"%c" -d @15996137957
Thu 30 Jul 2020 01:39:17 PM MDT
Get a file's last modified time in Unix timestamp stat -c %Y filename
Convert a multi-line list into a comma delimited string cat list | paste -s -d, -
Get all IPv4 addresses on this host ip a | grep -ohE 'inet [0-9.]{7,15}'
Get the URL from <a href="..."> echo '<a href="asdf">' | grep -Eo 'href="[^\"]+"' | sed 's/href="\(.*\)"/\1/'
Replace substring in Bash
Use ${variable/search_term/replace_term}. Eg:
Path="/etc/resolv.conf"

# This returns /etc/hostname
echo ${Path/resolv.conf/hostname}

Useful scripting patterns

Get a specific column

You can use either awk with delimiters specified with -F. 

$ echo one two three four five | awk '{print $3}'
three

$ echo one,two,three,four,five | awk -F, '{print $3}'
three

Or cut by a specific delimiter and select a specific column. This method cannot handle columns separated by more than one space. If you have neat columns, you'll need to pipe through tr -s ' ' first which collapses (or squeezes) the given repeating characters down. 

$ echo one two three four five | cut -d ' ' -f3
three
$ echo "one    two    three    four     five" | tr -s ' ' | cut -d ' ' -f3
three

$ echo one,two,three,four,five | cut -d ',' -f3
three

Or use the bash built-in function read which can take in a list of variables. Variable assignment by piping to read works in zsh but will only work if used within a subshell in bash. Keep this behavior in mind if you choose to use this method of separating out words in a string. For example:

## Basic variable assignment with read
$ echo one two three four five | while read _ second _; do echo $second ; done
two

## zsh variable assignment works without a subshell/loop
% echo one two three four five | read _ second _
% echo $second 
two

The last variable contains all remaining columns. Unwanted columns can be named repeatedly with _ (like in golang) and ignored.

## Remember, to get only the 3rd column, we need to read the 4th column out.
% echo one two three four five | read _ _ third; echo $third
three four five
% echo one two three four five | read _ _ third _ ; echo $third
three

## Delimiters specified with IFS
% echo one,two,three,four,five | IFS=',' read _ _ third _ ; echo $third
three

In terms of speed, be aware that using awk or tr+cut are actual commands that spawns a sub-process whereas read is a bash built-in function. As a result, it's always faster to use read to parse out data, especially if in a loop. If you are looping and using awk or cut, perhaps consider replacing it with the bash read equivalent. For example: 

## Super slow! Invokes awk 2 times per line
$ w | tail -n +3 | while read i ; do
  Username=$(echo "$i" | awk '{print $1}')
  Source=$(echo "$i" | awk '{print $3}')
  echo $Username $Source
done

## Instead do something like this!
$ w | tail -n +3 | while read Username _ Source _ ; do
  echo $Username $Source
done

Preserve header while sorting without using a temporary file

You can sort an output on a specific column while preserving the header and without needing a temporary file by using a shim function that reads and prints the first line before passing the rest of the output down the pipe. Define this function: 

# print the header (the first line of input)
# and then run the specified command on the body (the rest of the input)
# use it in a pipeline, e.g. ps | body grep somepattern
body() {
    IFS= read -r header
    printf '%s\n' "$header"
    "$@"
}

Then use this shim body function with the desired sort function. For example: 

$ top -b -n 1 | head -n 15 | tail -n +7 \
  | body sort -k 5 -rn

Credit to https://unix.stackexchange.com/questions/11856/sort-but-keep-header-line-at-the-top for this neat trick.

Testing things in Bash

test is a bash built-in function to add conditionals in your script. [ is an alias of the test command and are analogous.

Use the test operators listed below with test or a single bracket test condition like the following example. 

$ test -e /etc/hosts && echo "File exists"
File exists

## Note that the white space around [ and ] and also the operator are required.
$ [ -e /etc/hosts ] && echo "File exists"
File exists

$ if [ -e /etc/hosts ] ; then echo "File exists" ; fi
File exists
Operator Description
-e file exists
-a file exists (deprecated)

This is identical in effect to -e

-f file is a regular file (not a directory or device file)
-s file is not zero size
-d file is a directory
-b file is a block device. Eg. /dev/sda.
-c file is a character device. Eg. /dev/tty0.
-p file is a pipe. Eg. /dev/fd/0.
-h file is a symbolic link
-L file is a symbolic link
-S file is a socket
-t file (descriptor) is associated with a terminal device

This test option may be used to check whether the stdin [ -t 0 ] or stdout [ -t 1 ] in a given script is a terminal.

-r file has read permission (for the user running the test)
-w file has write permission (for the user running the test)
-x file has execute permission (for the user running the test)
-g file or directory has set-group-id (sgid) flag set

If a directory has the sgid flag set, then a file created within that directory belongs to the group that owns the directory.

-u file has set-user-id (suid) flag set
-k file or directory has sticky bit set
  • If set on a file, that file will be kept in cache memory, for quicker access.
  • If set on a directory, it restricts write permission. Setting the sticky bit adds a t to the permissions on the file or directory listing. This restricts altering or deleting specific files in that directory to the owner of those files.
-O you are owner of file
-G group-id of file same as yours
-N file modified since it was last read
f1 -nt f2 file f1 is newer than f2
f1 -ot f2 file f1 is older than f2
f1 -ef f2 files f1 and f2 are hard links to the same file
! "not" -- reverses the sense of the tests above (returns true if condition absent).

Integers

Integer tests, with test or inside single brackets.

Eg. 

$ test 2 -eq 2 && echo equal


Operator Description
-eq is equal to
-ne is not equal to
-gt is greater than
-ge is greater than or equal to
-lt is less than
-le is less than or equal to


Integer tests can also be done inside double parentheses. Eg. 

$ ((2==2)) && echo equal


Operator Description
== is equal
< is less than
<= is less than or equal to
> is greater than
>= is greater than or equal to


Strings

Operator Description
-z string is null, that is, has zero length
-n string is not null.
= is equal to 
## Does not work in zsh. Works only in sh and bash.
$ if [ "$a" = "$b" ] ; then
  echo "Equal"
fi
== is equal to 
## Does not work in zsh. Works only in sh and bash.
$ if [ "$a" == "$b" ] ; then
  echo "Equal"
fi

This operator inside a double bracket [[ ]] construct does pattern matching.

!= is not equal to 
$ if [ "$a" != "$b" ] ; then
  echo "Not equal"
fi

This operator inside a double bracket [[ ]] construct does pattern matching.

=~ Match using regex 
$ if [ zone36-ta =~ .*ta$ ] ; then
  echo "matches"
fi

$ if [ zone36-ta =~ *ta ] ; then
  echo "matches"
fi


Command Substitution

Command substitution allows output from a command to be substituted in place of the command itself in a script. Commands can be enclosed within backticks `command` or the POSIX compatible form $(command). 

## Eg.
echo "Today is " $(date)
echo "Today is " `date`

Do not confuse $(command) with $((expression)) which is used for arithmetic expansion.

Nested Substitution

If you are trying to do something similar to: 

Something=`basename `ls /home/*.txt``

You will get an error since the expansion is ambiguous. Instead, use the $(command) substitution instead: 

Something=$(basename $(ls /home/*.txt))


Arithmetic Operations

Do integer arithmetic operations on variables within double brackets. For example: 

Value=1
ValuePlusOne=$((Value + 1))
ValueTimes12=$((Value * 12))

The operators that are supported are: 

+ - / * %

Bitwise operators also work: 

^ (XOR)   | (OR)   & (AND)   << (Shift left)   >> (Shift right)

Control Blocks

Switch

read -p "What do you want to do?: " Option

case "$Option" in
	"K")
		install_new_kernel
		die "Complete!"
		;;
	"H")
		usage
		;;
	*)
		die "Stopping installation."
		;;
esac

See Also

Various Linux Notes
Linux Tools and Utilites
Package Management
Linux Distributions
Networking
Containers