Command Line Utilities

This is a collection of random useful command line tools.

Network Related

rsync

• --no-p: this will disable permissions synchronization.
• --no-o: this will disable owner synchronization.
• The -C option will exclude default files and directories like .svn, .cvs.
• By default rsync does not delete absent files on target directory; you must add --delete to do so.
  • If you need to have directories where the --delete option, and some protected (without that option) on the same rsync invocation, you need to use the --filter="protect /my-dir/*" rule.
• You can only have one rule per --filter option, if you need more, you just use more command line options or read the rules from a file.
• There is no option to specify a base directory for the paths in rsync; you just have to cd to that directory
• The -R (relative path) option can be very useful if you want to sync multiple directories at once. You can use the special syntax "/./" to specify which part of the path should go onto the target directory (read the manual page for more details).

Checking Algorithm

• rsync first looks at all the files whose size or modification time differ. Then it will run a MD5 algorithm on them to see if they differ and transfer the differing parts. You can give an option to only check by filesize (--size-only), ignoring modification times. You can also (I think) just look at all the files and run the MD5 on all of them (this can be slow since it implies a lot of I/O both at source and target).

• Note that if several people synchronize files via SVN and then use rsync to deploy to production, then it's better to disable modification time checking as the modification times will always change. So use "--size-only" with the "-O" option for directories (this will also disable modification time checking for directories).

Build system with rsync and Bash

• If you need to have a build system when you mainly need to copy and synchronize files between a development directory and a production one, rsync with Bash can be used to produce simple scripts that accomplish that goal. For instance with PHP development you usually need this kind of build system at some point.
• Here is an example of a possible script (here it's for cf-engine):

BASE_DIRECTORY="/home/elvanor/development/com.shoopz/overmind/"
TARGET_DIRECTORY="/srv/net.kameleoon-dev/overmind/"

cd "${BASE_DIRECTORY}"
rsync -avzCR --delete --no-owner --no-group --no-perms --filter="P public/*" --filter="P bootstrap/configuration/*.cf" "./public" "./cfengine" "./configuration" "./bootstrap" "./bootstrap-client" "development/./" "${TARGET_DIRECTORY}"

cp ${TARGET_DIRECTORY}cfengine/cf-execd.cf ${TARGET_DIRECTORY}cfengine/failsafe.cf ${TARGET_DIRECTORY}bootstrap/configuration/ && chown -R elvanor:swarm ${TARGET_DIRECTORY}bootstrap/ ${TARGET_DIRECTORY}bootstrap-client/
    && chmod -R g+w,o-rw ${TARGET_DIRECTORY}bootstrap/ ${TARGET_DIRECTORY}bootstrap-client/

nmap

• This program allows you to test if a port is open on a remote box. You may need the -Pn option to get it working.

nmap -Pn -p 53 176.31.246.207

• To check an UDP port (TCP is default), add the -sU option.

 nmap -Pn -sU -p 53 www.elvanor.net

• To list all devices on a given subnetwork:

nmap -sP 192.168.0.*

netstat

• netstat -altp will list all the open ports on your machine, along with the process that opened it. Very useful!
• The -n option will print the numeric values of ports, instead of resolving to symbolic names of services.

route

• The -n option will print IP addresses instead of symbolic server names.

File & I/O

df

• To obtain the space available on a HD: df -h

du

• To obtain a list of the sizes of all the directories of the current directory:

du --max-depth=1 -h

or

du -sh *

tar

• Use the -C option to switch to a directory before compressing. Very useful.
• Use the -h option to follow symbolic links.
• There does not seem to be an option to compress files from different base directories but keep the paths relative to the tarball. One alternative is to use tar several times with the --append option (you cannot use compression then). Example:

tar -C /home/elvanor/mediawiki/ -cf wiki.tar images
tar -C /tmp/ --append -f wiki.tar wiki.sql
bzip2 wiki.tar

cpio

• This seems to be something similar to tar.
• Allows you to do nice stuff like

find . -name '*.txt' print0 | cpio -o -0 | ssh elvanor@otherbox 'cd dirB && cpio -iduv -0'

This would copy the *.txt files to another box, preserving the directory hierarchy.

grep

• grep can be used to find all the occurences of a string in a directory:

grep searched_string -r /var/target/directory

find

• find can be used to find files containing a pattern in their names:

find . -name "*foo*"

Remote Control

screen

• Use control + a + d to detach; screen -r to reattach.
• Use screen -x to attach to a screen started by someone else. This allows you to share your screen.
• screen -x without arguments lists you all the screens running on a box.

Administration

gpasswd

• This is the command used on Gentoo to change an user's groups. You can also edit /etc/group, but this is not the right way of doing things. It will usually fail because you also need to change /etc/gshadow. Note that /etc/group- is just a backup of /etc/group.

useradd

• Command to add a new user.

Other

ps

• The legend for the status letters of processes can be found in the man page. R basically means running, and S means sleeping.
• Something very tricky can happen with ps: ps lists a process, but ps | grep will not. This is because if processes are always spawned and short lived (which means you have a problem on your system), ps when started on a shell will get the time to catch the process but ps | grep will run faster (less text to output) and so won't catch the process. A way to check for signs of this problem is to launch htop and see if the CPU is at 100% usage.
• ps can display the environment of the current running processes, with the e option.

pstree

• A variant of ps to display process in a tree hierarchy. Can be useful.

detox

• This small utility can sometimes be useful to rename files with corrupted or strange characters in the file name.

Tricks and tips

• Redirecting all outputs to a file:

foo &> bar

To redirect only stderr:

foo 2> bar

• To output to a file in addition to the console, use the tee binary:

echo "Hello World" | tee hello.txt
echo "Hello World" 2>&1 | tee hello.txt # This redirects ALL outputs to tee

• To find a file recently modified:

echo "foo" > /tmp/bar
find / -newer /tmp/bar