Bash Scripting

General

Command Line arguments conventions

• Long options are represented by two starting hyphens. Long options and short options should be provided before any other arguments.

Subshell

• To launch a command in a subshell, use (). It's usually used with $(), since that allows a variable to capture stdout of the launched process. Note that using backticks is deprecated.
• Note that using $() will trim newlines at the end of commands!

Standard Input / Output

• "<<<" can be used to feed a string as standard input.

Debugging

• set -x can be very useful for debugging Bash scripts. set +x returns to normal mode.

String Manipulations

• To replace all substrings by another, use the following syntax:

echo ${stringZ//abc/xyz}

This would replace all occurences of abc in stringZ by xyz. The following replaces only the first match:

echo ${stringZ/abc/xyz}

To replace something at the end of a string, use:

echo ${stringZ/%abc/XYZ}

Heredocs

• You can use << to start a heredoc. You must supply a limit string (EOF for instance). Note that you can still indent the heredoc; in this case use <<- which will strip the leading tabs of every line.
• You can use expansion inside a heredoc (via $) and even expand inside an arithmetic context: $(( )).
• You don't need to quote variables inside heredocs to preserve special characters like newlines. You actually should not quote as the " symbol is then present into the heredoc (which of course you don't want).

printf

• printf can be very useful when you need to insert newlines into strings. Example:

printf -v deleteCommand "%sd\n%d\n" "${deleteCommand}" "${i}"

• The -v option stores the result into a variable.

Built-ins

• The exec built-in allows you to start a process with the same pid as the current Bash process id. This can be useful sometimes; the new process takes the pid of the old one.

Control structures

Loops

• To easily loop over all the files in a directory, you can just use parameter expansion:

for file in *; do echo $file; done

• The following command can be used to repeatedly convert files for instance:

for f in ./*.svg; do inkscape "$f" -z -A "pdf/${f%.svg}.pdf"; done

• C style loops are possible:

for ((i=1; i <= partitionsNumber; ++i))

• Loop over an array:

for i in "${myArray[@]}"
do
	echo $i
done

Tests

• The then keyword is mandatory after if / fi.
• [ is not a keyword but a command (a program!). It is recommended to use [[ in tests which is a keyword.
• Note that [[ myVariable ]] will output true even if myVariable is equal to 0. The test must be explicit.
• After [[ and before ]] there must be a space character.

General

• -eq is used for checking equality, -neq for the contrary.
• -n checks if a string is not empty, -z if it is empty.

if [[ -z $(echo "$output" | grep "1920x1080" | grep "30.00\*") ]]
then
    echo "Hello World"
fi

Strings

• "==" can be used inside [[ to check for string equality.

Arithmetic Context

• "!=" can be used inside (( )).
• I am not sure of the interest of arithmetic context for tests. All tests I've seen seems to work with [[ ]].

Files

• Tests for files.

Variables and parameters

Special Symbols

• "$@" expands to all command-line parameters.
• "\n" in a variable does not necessarily works as expected. Eg, no newline is created.
• \ before a newline actually escapes the newline. Thus, you can create multi-line strings or commands just by terminating a line with the \ symbol.
• Positional parameters can be accessed with $1, $2 etc. You only need brackets {} after the 9th parameter ("${10}").
• $? stores the exit status of the last ran command. So echo $? is useful to see the exit status of the previously ran command.

Quoting

• If you need to expand special characters such as *, you cannot quote. Don't forget that you can quote partially.
• Always try to quote, as if you don't quote newlines for instance are changed to whitespaces. An exception is in heredocs, as the doublequotes will stay in the heredoc.
• Use single quotes rather than double quotes, especially in sed. If you use double quotes, the \ itself won't be taken as a \ for escaping, thus causing problems.
• If you write \ and then a newline, the newline will be escaped which allows you to write multiline strings.
• Bash has a special quoting mode for strings: $'my literal string'. This will allow newlines to be inserted using "\n", but you cannot use variable expansion.

Expansion

• The shell expands stuff like aaa* as soon as it sees this expression. Thus if you define a custom function myFunc(), and call it like that:

myFunc stuff*

If there are two files stuff1 and stuff2 in the current directory, $1 and $2 will be set to stuff1 and stuff2. Even if they are quoted ("$1", "$2") since the expansion took place before.

• If you write a command like:

MYVAR="35" echo ${MYVAR}

the output is not the expected 35. This is because MYVAR expansion happens before echo is launched with the MYVAR variable set on the environment.

Parameters

• Parameters can be "moved" using the shift keyword, which can be very useful when parsing all parameters given to a script. Warning however: if you use shift 2 for instance and there is only one parameter, this will silently fail and not move anything at all.

Arrays

• You can declare an array like this:

declare -a arrayname=(element1 element2 element3)

• The length of an array can be obtained via "${#array[@]}".

Arithmetic context

• When within double parenthesis (( )) or after the let keyword, Bash enters arithmetic context. You don't need to quote variables or precede them with a $ sign. Tests also work as expected, eg more like in their C counterpart. For example,

if (( myVariable )) 

will return false if myVariable is equal to 0.

• Arithmetic context also applies when in an array [].

Functions

• Functions must be declared before they are called. You call a function without the () (eg., just the function name).
• The local keyword declares a variable local to a function; if you wish to declare several local variables, just separate them with spaces (not commas).

Command Line Utilities

• sdiff -s will generate a formatted output of the differences between two files. Very useful.

sed

• sed is a stream editor (help here). It is extremely powerful and can do almost everything under the sun. It can:
  • delete a line with command d;
  • append with command a;
  • use multiline strings if needed, with the standard Bash mechanism;
  • search and replace a string by another (command g); be careful with this: you cannot output to the same file you read from, it will produce a blank file!
  • use several replacements on one line. In this case you need to add the option -e to all changes (else sed will only take one argument for replacement). Example:

sed -e "/dir=\"plugins\/org.eclipse.jdt.apt.pluggable.core\"/d" \
	-e "/dir=\"plugins\/org.eclipse.jdt.compiler.apt/d" \
	-e "/dir=\"plugins\/org.eclipse.jdt.compiler.tool\"/d" \

• With the \1 option, you can use sed to search in a string relatively simply. It outputs the result of the group in the regular expression:

ifconfig eth0 | grep "inet addr" | sed "s/.*inet addr:\([0-9\.]*\).*/\1/"

• sed can handle multiple lines (a complete file), but it does not seem very easy, so in this case another tool is probably better.

grep

• grep is normally used on lines (via stdin), but it can also be used for files if given as arguments. If you use the -l switch, the normal output is disabled and it just prints the path to the files where output would have been printed.

find

• find is a tool to find given files on a filesystem according to certain criteria.
• Every option given to find is a predicate. For instance:
  • -type d: will only find directories
  • -exec bash: will load bash and consider the file if the bash invocation returned 0

• Example to find all files with "conflict" in their names:

find . -name '*conflict*' -print

• If you want to delete those:

find . -name '*_conflict-*' -print -delete

• Same with modification dates:

find . -name '*conflict*' -printf "%p %TY-%Tm-%Td %TH:%TM:%TS %Tz\n"

• Complex example, which will print out all subdirectories of /var/db/pkg/ that do NOT contain a file named CONTENTS:

find /var/db/pkg/ -type d -exec bash -c '! -e "$1/CONTENTS" ' -- {} \; -print

• bash is invoked with 4 arguments, -c, then the command string, then -- as $0 and then {} (which corresponds to the canonical file path given by find) as $1.

chroot

• A mandatory command should be given to chroot; it represents the command that will be executed inside the chroot environment. If you wish to chroot inside a Bash script, you should copy over a script to the new chroot and execute it.

Techniques

• To rename all the files in a folder to lower case:

for f in $(find -maxdepth 1); do mv -v $f $(echo $f | tr '[A-Z]' '[a-z]'); done

• To set all files readable by everyone, writable only by user, and set the executable bit only on directories (which is usually what you want, except for executables inside those directories):

chmod -R a-x+rX,u+rw,go-w,go+r *