Bash Scripting

General

Setup

• To get universal history on all your terminal tabs, as well as instant appending to history (rather than default behavior which appends to disk only when session is closed), add those two lines to your ~/.bashrc:

shopt -s histappend                      # append to history, don't overwrite it
export PROMPT_COMMAND="history -a; history -c; history -r; $PROMPT_COMMAND"

Command Line arguments conventions

• Long options are represented by two starting hyphens. Long options and short options should be provided before any other arguments.

Subshell

• To launch a command in a subshell, use (). It's usually used with $(), since that allows a variable to capture stdout of the launched process. Note that using backticks is deprecated.
• Note that using $() will trim newlines at the end of commands!

Standard Input / Output

• "<<<" can be used to feed a string as standard input.

Debugging

• set -x can be very useful for debugging Bash scripts. set +x returns to normal mode.

String Manipulations

• To replace all substrings by another, use the following syntax:

echo ${stringZ//abc/xyz}

This would replace all occurences of abc in stringZ by xyz. The following replaces only the first match:

echo ${stringZ/abc/xyz}

To replace something at the end of a string, use:

echo ${stringZ/%abc/XYZ}

Heredocs

• You can use << to start a heredoc. You must supply a limit string (EOF for instance). Note that you can still indent the heredoc; in this case use <<- which will strip the leading tabs of every line.
• You can use expansion inside a heredoc (via $) and even expand inside an arithmetic context: $(( )).
• You don't need to quote variables inside heredocs to preserve special characters like newlines. You actually should not quote as the " symbol is then present into the heredoc (which of course you don't want).

printf

• printf can be very useful when you need to insert newlines into strings. Example:

printf -v deleteCommand "%sd\n%d\n" "${deleteCommand}" "${i}"

• The -v option stores the result into a variable.

Built-ins

• The exec built-in allows you to start a process with the same pid as the current Bash process id. This can be useful sometimes; the new process takes the pid of the old one.

Control structures

Loops

• To easily loop over all the files in a directory, you can just use parameter expansion:

for file in *; do echo $file; done

• The following command can be used to repeatedly convert files for instance:

for f in ./*.svg; do inkscape "$f" -z -A "pdf/${f%.svg}.pdf"; done

• C style loops are possible:

for ((i=1; i <= partitionsNumber; ++i))

• Loop over an array:

for i in "${myArray[@]}"
do
	echo $i
done

Tests

• The then keyword is mandatory after if / fi.
• [ is not a keyword but a command (a program!). It is recommended to use [[ in tests which is a keyword.
• Note that [[ myVariable ]] will output true even if myVariable is equal to 0. The test must be explicit.
• After [[ and before ]] there must be a space character.

General

• -eq is used for checking equality, -neq for the contrary.
• -n checks if a string is not empty, -z if it is empty.

if [[ -z $(echo "$output" | grep "1920x1080" | grep "30.00\*") ]]
then
    echo "Hello World"
fi

Strings

• "==" can be used inside [[ to check for string equality.

Arithmetic Context

• "!=" can be used inside (( )).
• I am not sure of the interest of arithmetic context for tests. All tests I've seen seems to work with [[ ]].

Files

• Tests for files.

Variables and parameters

Special Symbols

• "$@" expands to all command-line parameters.
• "\n" in a variable does not necessarily works as expected. Eg, no newline is created.
• \ before a newline actually escapes the newline. Thus, you can create multi-line strings or commands just by terminating a line with the \ symbol.
• Positional parameters can be accessed with $1, $2 etc. You only need brackets {} after the 9th parameter ("${10}").
• $? stores the exit status of the last ran command. So echo $? is useful to see the exit status of the previously ran command.

Quoting

• If you need to expand special characters such as *, you cannot quote. Don't forget that you can quote partially.
• Always try to quote, as if you don't quote newlines for instance are changed to whitespaces. An exception is in heredocs, as the doublequotes will stay in the heredoc.
• Use single quotes rather than double quotes, especially in sed. If you use double quotes, the \ itself won't be taken as a \ for escaping, thus causing problems.
• If you write \ and then a newline, the newline will be escaped which allows you to write multiline strings.
• Bash has a special quoting mode for strings: $'my literal string'. This will allow newlines to be inserted using "\n", but you cannot use variable expansion.

Expansion

• The shell expands stuff like aaa* as soon as it sees this expression. Thus if you define a custom function myFunc(), and call it like that:

myFunc stuff*

If there are two files stuff1 and stuff2 in the current directory, $1 and $2 will be set to stuff1 and stuff2. Even if they are quoted ("$1", "$2") since the expansion took place before.

• If you write a command like:

MYVAR="35" echo ${MYVAR}

the output is not the expected 35. This is because MYVAR expansion happens before echo is launched with the MYVAR variable set on the environment.

Parameters

• Parameters can be "moved" using the shift keyword, which can be very useful when parsing all parameters given to a script. Warning however: if you use shift 2 for instance and there is only one parameter, this will silently fail and not move anything at all.

Arrays

• You can declare an array like this:

declare -a arrayname=(element1 element2 element3)

• The length of an array can be obtained via "${#array[@]}".

Arithmetic context

• When within double parenthesis (( )) or after the let keyword, Bash enters arithmetic context. You don't need to quote variables or precede them with a $ sign. Tests also work as expected, eg more like in their C counterpart. For example,

if (( myVariable )) 

will return false if myVariable is equal to 0.

• Arithmetic context also applies when in an array [].

Functions

• Functions must be declared before they are called. You call a function without the () (eg., just the function name).
• The local keyword declares a variable local to a function; if you wish to declare several local variables, just separate them with spaces (not commas).

Command Line Utilities

• sdiff -s will generate a formatted output of the differences between two files. Very useful.

sed

• sed is a stream editor (help here). It is extremely powerful and can do almost everything under the sun. It can:
  • delete a line with command d;
  • append with command a;
  • use multiline strings if needed, with the standard Bash mechanism;
  • search and replace a string by another (command g); be careful with this: you cannot output to the same file you read from, it will produce a blank file!
  • use several replacements on one line. In this case you need to add the option -e to all changes (else sed will only take one argument for replacement). Example:

sed -e "/dir=\"plugins\/org.eclipse.jdt.apt.pluggable.core\"/d" \
	-e "/dir=\"plugins\/org.eclipse.jdt.compiler.apt/d" \
	-e "/dir=\"plugins\/org.eclipse.jdt.compiler.tool\"/d" \

• With the \1 option, you can use sed to search in a string relatively simply. It outputs the result of the group in the regular expression:

ifconfig eth0 | grep "inet addr" | sed "s/.*inet addr:\([0-9\.]*\).*/\1/"

• sed can handle multiple lines (a complete file), but it does not seem very easy, so in this case another tool is probably better.

grep

• grep is normally used on lines (via stdin), but it can also be used for files if given as arguments. If you use the -l switch, the normal output is disabled and it just prints the path to the files where output would have been printed.

find

• find is a tool to find given files on a filesystem according to certain criteria.
• Every option given to find is a predicate. For instance:
  • -type d: will only find directories
  • -exec bash: will load bash and consider the file if the bash invocation returned 0

• Example to find all files with "conflict" in their names:

find . -name '*conflict*' -print

• If you want to delete those:

find . -name '*_conflict-*' -print -delete

• Same with modification dates:

find . -name '*conflict*' -printf "%p %TY-%Tm-%Td %TH:%TM:%TS %Tz\n"

• Complex example, which will print out all subdirectories of /var/db/pkg/ that do NOT contain a file named CONTENTS:

find /var/db/pkg/ -type d -exec bash -c '! -e "$1/CONTENTS" ' -- {} \; -print

• bash is invoked with 4 arguments, -c, then the command string, then -- as $0 and then {} (which corresponds to the canonical file path given by find) as $1.

chroot

• A mandatory command should be given to chroot; it represents the command that will be executed inside the chroot environment. If you wish to chroot inside a Bash script, you should copy over a script to the new chroot and execute it.

Techniques

• To rename all the files in a folder to lower case:

for f in $(find -maxdepth 1); do mv -v $f $(echo $f | tr '[A-Z]' '[a-z]'); done

• To set all files readable by everyone, writable only by user, and set the executable bit only on directories (which is usually what you want, except for executables inside those directories):

chmod -R a-x+rX,u+rw,go-w,go+r *