code-backups/bash/shell-scripting-notes

Shell scripting notes

These are just Unix/Linux shell scripting notes. Even though it's named "scripting" notes, there are many things listed here which can be used for using the terminal better. It is mostly written for bash/ksh. But at least some, possibly all, can be applied to /bin/sh as well.

NOTE: Since these are just personal notes it might contain some mistakes. If you find any, please post an Issue or a PR. Research yourself before adapting anything.

Outline

a. Shell Basics b. Running things c. Handling outputs d. Variables

Shell Basics

You can run shell commands on a terminal application, such as, xterm, LXTerminal, sakura, st, GNOME Terminal and even on Termux (Android). You can also run these on a TTY (you can access a TTY by pressing Ctrl+Alt+F1 through F8 and logging in).

If you login as a normal user on TTY or just open a GUI terminal, you may see a $ sign on your prompt. If you login as root, you may see a #. Even though there are exceptions, this is fairly common in most systems.

Following this convention, commands that are supposed to be run as normal user starts with a $ at the beginning of the line. Otherwise if it is meant for the root user to run, it starts with a # at the beginning:

$ echo test
test
$ whoami
john
$ su
# whoami
root

As you can see, lines that do not have either $ or # are output lines. When we switched the user to root, the line is shown with # at the beginning. This is a not a hard-set rule and anybody can invent anything else. But it is more or less followed by everyone.

To know which user you are currently logged in as, run whoami and it will tell you. To get out of root user mode, run exit or press Ctrl+D.

To know which shell you are on, run either of these:

$ echo $0
ksh
$ echo $SHELL
/bin/ksh

In my experience $0 is more reliable, since $SHELL returns the initial shell even if subsequent shells are accessed.

This is just an example. If you are running something else, like bash, zsh etc. you will get output according to that.

Running multiple commands

From a terminal shell, it's as easy as using curly braces ({...}). If you want to run multiple commands:

1. Type { and enter
2. Type or paste in commands as you wish
3. When done, press enter to go into a new line
4. Type } and enter

This will execute all the commands you entered.

There are other ways to do this with advantages and disadvantages.

Running through a script file

There is a way to save the commands on a file. The filename should be preferably with .sh extension, but anything else even without extension would work as well.

There are 2 ways of running the file:

1. Something like bash test.sh.
2. If it has a shebang line like #!/usr/bin/env bash as the first line, the script can be made executable with chmod +x test.sh then run with ./test.sh or /path/to/test.sh.

Saving to a file makes it easier if you want to run the script again in future. You won't have to type each command again in order to run it.

Running things

Running something on background

Usually when we run something from a terminal it occupies the terminal. We cannot do anything else until the program terminates. To run something and continue to use the terminal this can be used:

someprogram &

But this has some problems. It lets us use the terminal for other things but:

1. The program terminates when the originating terminal window closes.
2. And if there is any output generated by the program it is printed on the originating terminal while it is being used for something else, which can be really annoying.

So there are some ways to deal with it: nohup or disown

In systems where there nohup exists, it can be used:

nohup someprogram

It may print something like "sending output to nohup.out" and nothing else, so the terminal can be freely used for anything else or even exited.

Although it creates a nohup.out file with the program output on current directory or if not possible, on $HOME.

If it is available, you can also use disown:

someprogram & disown

Even though it works, it outputs everything on originating terminal. So:

someprogram >/dev/null 2>&1 & disown

> without any number (e.g. 1>) means 1. &1 is referring to 1> which is /dev/null here. So these three does the same thing:

someprogram 1>/dev/null 2>/dev/null & disown
someprogram >/dev/null 2>/dev/null & disown
someprogram >/dev/null 2>&1 & disown

To avoid repitition we use &1.

Further explanation of redirection methods can be found here. &>/dev/null might not work everywhere except bash.

On script files, we can safely ignore nohup or disown or similar solutions.

Capturing Process ID

To run something that you want to kill later, you can capture the process id (PID) so that a kill signal can be sent to it. $! is a special variable that returns the PID of last background run job:

$ for i in `seq 1 10`; do echo $i; sleep 10; done &
$ PID=$!
$ kill -3 $PID

Here, we are running for loop that prints a number from 1 to 10, with 10 seconds break in between. It will take more than a minute to finish this. If immediately after running this, we run the following 2 commands, it will kill the for process and stop the counting.

Some programs offer specifying a PID file for this exact reason. PID file is just a regular file that contains the PID as the file content. e.g. tor has a PidFile option.

Program exit code

Each program is meant to return exit code of 0 (zero) when it runs and ends successfully. Anything other than that means there was something wrong.

$? is a special variable that contains the exit code for last run command.

$ ls /tmp
$ echo $?
0
$ ls nonexistingfile.txt 
ls: nonexistingfile.txt: No such file or directory
$ echo $?                
1

The last ls command had an error, so it returned 1. We can use this to show messages, take actions and what not:

#!/usr/bin/env bash

ls /root
if [ "$?" = '0' ]; then
	echo 'accessing root files was successful'
else
	echo 'accessing root files has failed!'
fi

You can save the above on a test.sh file and run bash test.sh. On an adequately secure system, normal users shouldn't be able to ls root files. Only root user should. So, depending on which user you run this as, it will either show a success or a failure message.

Above can be shortened with:

#!/usr/bin/env bash

if ls /root; then
	echo 'accessing root files was successful'
else
	echo 'accessing root files has failed!'
fi

To keep things simple, the output wasn't suppressed. ls /root >/dev/null 2>&1 can be used instead of ls /root to suppress output.

Handling outputs

Keeping outputs in files

Output produced by a program can be put in a file by:

someprogram > output.txt

No matter what someprogram prints out, it will be saved to output.txt. e.g.

ls -la > listoffiles.txt

After running this listoffiles.txt will have the list of files and directories in listoffiles.txt.

However, > will delete previous content of the output file. If you want to keep the existing content of the output file and just add to it, use >> instead of >. e.g.

date >> date.txt
date >> date.txt
date >> date.txt

After running the first one it will add a line to date.txt, and after running second one it will add to it. These 3 commands will make the date.txt file have 3 dates, each on it's own line. This means it didn't delete previous content.

There is another way to add or append something to a file, and that's tee.

echo test | tee -a test.txt

Each time you execute this command, a line "test" will be added to test.txt.

tee is extremely helpful when you want to add to a file in a priviledged location. e.g.

echo 'someconfig = 1' | sudo tee -a /etc/sysctl/someconfig.conf

tee can be run without -a to replace existing content. e.g.

echo test | tee test.txt

Replaces existing content and places "test" to the test.txt.

Keeping output in variables

We can store the output of a program in a variable like this:

$ mydate=$(date)
$ echo $mydate
Tue Mar 22 20:04:42 +00 2022

With $(date) we are capturing the output of date command into a variable. We are then setting that variable value to mydate variable.

Another example:

$ usrlist=$(ls /usr)
$ echo "My /usr contains:\n$usrlist"
My /usr contains:
X11R6
bin
distfiles
games
include
lib
...

$(...) and `...` does the same thing. You can try the above example with usrlist=`ls /usr` and it should do the same thing. For clarity and consistency, use either one in a project and stick with it. I like $(...) because it's easier to read and find. But the other one is ok too if you want to use it.

Variables

Basics

Keeping something in a variable is as easy as doing:

a=5

Now if you do a echo $a it should return "5".

This is a small example so it might not need quotes. With values containing spaces it will need a quote. e.g.

name='John Doe'

Now echo $name will work. But it is a good practice to surround variables with double quotes. A double quote takes care of whether the string is one word or multiple words or even multiple lines (especially when using on if conditions). So, echo "$name" would be perfect.

Using $ to refer to a variable only works inside a double quote. It does not work under single quotes.

$ echo "$name"
John Doe
$ echo '$name' 
$name

Most of the time double quote is what we want, unless we need to print the variable name specifically, e.g. in an error message containing the variable name.

To add something to a variable, double quote is the way to go. e.g.

echo "My name is $name"

This is ok for most cases if you don't need any letters, underscores (_) directly after the name. But if you do need to place one, you'd have to be specific of the variable name:

$ echo "My name is ${name}_tnrkdnmk"
My name is John Doe_tnrkdnmk

If you add ${...} around the variable name, you can add anything after it.

Concatenating

Adding one string with another (aka concatenating) is possible:

$ firstname=John
$ lastname=Doe
$ echo "$firstname $lastname"
John Doe
$ echo $firstname $lastname   
John Doe
$ echo $firstname$lastname  
JohnDoe
$ echo "some""text"
sometext
$ echo $firstname", the neighbor"
John, the neighbor

Multiline strings

Strings with multiple lines are as easy as:

~ $ multi="one line\nanother line"
~ $ echo $multi
one line
another line

\n inside double quotes makes a new line.

Looping through multiple lines is easy:

$ echo "$multi" | while IFS= read -r line; do echo "Reading line:" $line; done
Reading line: one line
Reading line: another line

Further reading

• 🖌️ BASH Style Guide - 🎨 BASH Style Guide (my 2¢) by @bex