Wiki

Login
Search:

- Login
- Comments
- Info
- Attachments
- More Actions:

#

- ShellTricks

Translation(s): English - Français - Italiano | ?Discussion

Contents

1. Shell Scripting Tricks
   1. How do I see how much disk space is left?
   2. How do I see where all my big files are? My file system is full.
   3. I have a directory full of MP3s... how do I rename them all with underscores instead of spaces?
   4. How do I do that recursively?
   5. How do I (recursively) rename all files from uppercase to lowercase?
   6. How do I remove a file that begins with "-"?
   7. How do I monitor something in realtime?
   8. How do I get only the filename in a full pathname?
   9. How do I test whether a directory has files in it?
   10. How do I start a process in background?
   11. How do I work with arrays in bash?
   12. How do I use a variable within a variable (variable interpolation) in bash?
   13. How do I count the lines of files with certain extensions in a certain directory?

# Shell Scripting Tricks

Note: Debian Wiki is not necessarily the best place to find help with shell scripts. If your scripting problem involves bash (or Bourne shell), you might try bash instead. A bash FAQ is at http://mywiki.wooledge.org/BashFAQ and contains many more tricks than this page does.

## How do I see how much disk space is left?

    df
    df -h # for a human readable format in megs & gigs

## How do I see where all my big files are? My file system is full.

    cd /somewhere
    du -sk * | sort -n
    # Repeat as necessary

If you want to see big directories instead of files, you can do:

    du -x /somewhere | sort -n | tail -10
    # This shows the 10 largest directories under /somewhere

Another way:

    find /somewhere -size +2000k -ls
    # This shows all files over 2000 kilobytes under /somewhere

If you want to keep your .debs, you can filter them out of the list:

    find / -size +2000k ! -name "*.deb" -ls
    # This shows all files on your entire system over 2000 kilobytes that are not debian packages

## I have a directory full of MP3s... how do I rename them all with underscores instead of spaces?

    rename 's/ /_/g' *.mp3

The `rename` command is not a general Unix command but it's included with Perl which is of course installed by default in Debian.

## How do I do that recursively?

That's a bit tricky. You can get a list of files to pass to `rename` using `find`, but if some of the directories are being renamed as well, that's not something `rename` can keep track of. You have to use `-depth` with `find` to make sure any files are renamed before the directories they're in.

    cd /somewhere
    find . -depth -name '* *' -type f -print0 | xargs -r0 rename 's/ /_/g'

## How do I (recursively) rename all files from uppercase to lowercase?

Once again, this is hard if the directories themselves have uppercase letters in their names. Let's assume for the moment that they don't. Then:

    cd /somewhere
    find . -name '*[A-Z]*' -type f -print0 | xargs -0 rename 'y/A-Z/a-z/'

Unfortunately, whenever this question pops up, that's usually not enough -- the person asking usually has uppercase directory names too. So something like this might be required:

    cd /somewhere
    find . -type d -depth -name '*[A-Z]*' -print |
      while read dir; do dname="$(dirname $dir)"; bname="$(basename $dir)";
      newbname="$(echo $bname | tr [:upper:] [:lower:])"; mv "$dir" "$dname/$newbname"; done
    # That renames the directories.  The -depth makes it go through the deepest ones first, so we don't rename
    # A to a until we've already renamed A/B to A/b.
    find . -name '*[A-Z]*' -type f -print0 | xargs -0 rename 'y/A-Z/a-z/'

Note that the more complex script above is imperfect: it will fail if any directories have newlines in their names, and it may also fail under some conditions if any files or directories have whitespace in their names. Use at your own risk. (Hint: if you want to test it before you commit to it, put an `echo` in front of the `mv`.)

In fact, this simpler solution might work better:

    find /somewhere -depth -name '*[A-Z]*' -print0 | xargs -r0 rename 'y/A-Z/a-z/'

Here, too, it might be a good idea to put in an `echo` before the `rename` command to test before you actually run this.

## How do I remove a file that begins with "-"?

    unlink -foo

Three different ways:

    rm -- -foo
    rm ./-foo
    using 'mc', and pressing F8 on the appropriate file

These two tricks work with nearly every command line tool, not only `rm`.

## How do I monitor something in realtime?

If it's a log file, follow the tail of the file:

    tail -f /var/log/messages

or

    less +F /var/log/messages

If it's a general command, use `watch`:

    watch -n 1 ls -l ~/some/file

This works well when monitoring a firewall with iptables / tc or like tasks.

## How do I get only the filename in a full pathname?

Two ways:

    basename /path/to/file
    foo=/path/to/file ; echo ${foo##*/}

## How do I test whether a directory has files in it?

    if [ "$(ls -A somedir)" ]; then
        echo "It has files"
    fi

Note: a single argument inside the brackets is equivalent to `[ -n argument ]`. There seems to be no clean way to do this using only shell builtins. (We had one earlier attempt which was close, but that failed if there was one file named * in the directory.)

Another way:

    if [ "$(ls -A somedir | wc -l)" -gt 0 ]; then
        echo "found files"
    fi

A shorter way: `[ $(ls -A somedir) ] && echo "directory is not empty"`

Note: The double ampersand works like a boolean "and", so the "if" is not required.

## How do I start a process in background?

Just append an ampersand:

    sleep 100 &

It prints a line that looks like: `[1] 6338` , indicating that this process is job number 1 and that its process ID is 6338.

To kill this job:

    kill %1

or

    kill 6338

Note: If you have started many processes in background and you want to list them, type `jobs`.

You can also switch a job running in the background into the foreground, then suspend this job, and then restart this job to the background. Let's start a process in the background:

    sleep 100 &

To move it to the foreground (assuming that this is job number 1):

    fg %1

Then, press Ctrl+Z. This suspends the process and send it to the background. Now, restart the suspended process while leaving it in the background:

    bg %1

To kill this job:

    kill %1

## How do I work with arrays in bash?

Assigning to one:

    arr=(one two three four)
    myoggs=(*.ogg)

Iterating over one:

    for f in "${myoggs[@]}"; do ...; done

Selecting a random element:

    mysigs=($HOME/sigs/*)
    cat ${mysigs[RANDOM % ${#mysigs[*]}]}

(In the previous example, note that the brackets for the array index also force a numeric evaluation context, in which RANDOM doesn't need a leading $ sign.)

You can't easily "delete" an element from the middle of an array, but you can hack around it by reindexing the array to skip over all the unset elements:

    arr=(zero one two three four)
    unset arr[1]
    arr=("${arr[@]}")

This will also leave all the empty elements in place, deleting only the unset ones.

## How do I use a variable within a variable (variable interpolation) in bash?

If you want to do something like ${$var}, use this format: ${!var}

So if you have:

    FOO=one
    BAR=FOO

then ${!BAR} will return "one"

This only works for referencing an existing variable, not for assignment. If you wish to assign to a dynamically generated variable name, you must use either eval:

    IFACE=eth0
    eval IP_${IFACE}=192.168.1.1

or, in many cases, just use an array:

    N=17
    ROW[N]="This is the 18th row (counting from 0)."

The eval trick can also be used for referencing:

    IFACE=eth0
    eval echo \$IP_${IFACE}

The eval makes a second pass over the code. The first pass substitutes $ for \$ and then substitutes the value of ${IFACE}, leaving "eval echo $IP_eth0". The second pass then substitutes the value of the ${IP_eth0} variable, assuming one exists.

## How do I count the lines of files with certain extensions in a certain directory?

    find $dir -name *.[$extensions] -exec cat \{\} \; | wc -l

finds the files in $dir with any of the extensions $extensions and reads the files using cat using 'wc -l' to count the lines of text.

You can so make

    wc -l $dir/*.$extensions