C&CZ’s Not Frequently Enough asked questions

with answers!

Last modified: 2018-03-21

For questions: <postmaster@science.ru.nl>

Further reference: http://wiki.science.ru.nl/cncz/

About this FAQ

Contributors

  • Patrick Atoon patricka@cs.kun.nl
  • Kees Keijzers keesk@science.ru.nl
  • Jos Alsters josal@science.ru.nl
  • Peter van Campen Peter.vanCampen@science.ru.nl
  • Simon Oosthoek S.Oosthoek@science.ru.nl

Linux shells

What is a shell?

The shell is a program that interprets the user’s commands and executes them on the Operating System. There are many kinds of shells, which have different interpreted languages in which to write commands.

A shell can be used interactively, a human user typing commands on a terminal prompt, or in a shellscript. The shellscript is also called a batch file. A shell script that asks for human input can only be run from an interactive shell.

Examples of shells

On the linux machines you can use the shells:

  • /bin/sh : the default “Bourne” compatible shell, on Ubuntu, this is /bin/dash
  • /bin/dash: a light-weight Bourne/POSIX compatible shell.
  • /bin/bash: the Bourne Again Shell. A full fledged linux shell and the default.
  • /bin/zsh : the last shell you’ll ever need, very configurable
  • /bin/tcsh: a csh compatible shell. We strongly advise not to use this for scripts!

Note that there are shells for microsoft windows, e.g. cmd.exe and powershell. Recently the bash shell is becoming available on windows as well.

What is a command?

A command is either a built-in command of the shell, or a file with executable instructions in it.

The built in commands are executed by the shell, so they do not require the kernel to read another file and execute a different process. For this reason, some commands that are used often in scripts or on the commandline are not just available as an executable file, but also as internal command (e.g. echo and /bin/echo). A lot of built-in commands have to be built in, e.g. set or cd, because they act on the state of the shell itself.

Most commands are executable files, these can be separated in two categories; scripts and binary executables. Scripts are interpreted by an interpreter, like bash, python, perl, etc. Binary executables are programs compiled to machine code and stored in a format that the kernel understands.

How does the shell find a command?

Way to find the command Examples
an absolute path-name, starting with ‘/’ prompt> /usr/bin/vim or prompt> /usr/local/datasearch/bin/telgids
a relative path-name, containing at least one ‘/’ prompt> bin/testprogramma or prompt> ../petervc/bin/backup-photon
a name without ‘/’ built-in the shell prompt> echo “something” or prompt> read X
a name without ‘/’ found in the ‘$PATH’ prompt> ls or prompt> firefox

PATH

The variable PATH is important, it decides which file is run as command if not specified with an absolute or relative path-name. If multiple files with the same name are present on the system, the order of the directories in the PATH variable determines which of the files is executed.

prompt> echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:~/bin:~/bin

The PATH variable is a list of directories separated with a colon character. It is evaluated from left to right. Note that the current directory (‘.’) is not in the PATH and if you want to execute a file that is in your current working directory (see $PWD or the pwd command), you need to specify it using a relative or absolute path-name:

prompt> ./mycommand

Some people want to have commands in the current directory to be found in the PATH, which is possible, however it has risks, especially if you are working as superuser on a machine, but even as common user, you run the risk of executing a commandfile that some malicious user placed in a directory in which you are lured to cd to.

If you insist on doing this, please make sure you put the ‘.’ at the end of the PATH, so system commands cannot be hidden behind a command file in the current directory with the same name!

What is man

Most commands have a manual page, you can invoke the manual page by giving the command: man <command> in an interactive shell.

prompt> man bash

will give you the manual page of the bash shell. Manual pages have a few common headers which you can find in nearly all man pages. Once you get used to this, you can quickly find out how to use the command more effectively.

prompt> man -k something

Will search for the keyword something in the summary of each man-page on the system. This may help you find the right command to do what you want, without knowing the exact command name. This is not a fool-proof solution and often a search engine will be more effective!

Typical Linux commandline

If you work in a terminal (window) you can give commands via the shell. The shell interprets your command, most importantly it splits the line you provide into words separated by whitespace.

promp> COMMAND [arg1] [arg2] ... <enter>
item explanation
prompt> printed by the shell, usually user@host $, but it can be customised!
COMMAND the command you want to execute (you type it)
[arg1] argument for the command
[arg2] another argument for the command
… further arguments for the command
<enter> You press the return key, or enter, to “give” the command to the shell. After this, your line will be interpreted and executed.

examples

ls -l long (-l) listing of all files and directories in the current working directory

ls -a -l long (-l) listing of all files and directories in the current working directory, also show hidden entries (-a)

ls -l /tmp long (-l) listing of all files and directories in /tmp

ls -l /bin/bash long (-l) listing of the file /bin/bash

man man show manual page of the man command

What is the environment?

The environment is a collection of shell variables that is passed to the commands you type. Check what is in the environment with the command:

$ env

In a Bourne shell, a variable can be set like this:

prompt> MYVAR=something

to show what the variable holds (Note the $ sign):

prompt> echo $MYVAR

To make it part of the environment:

prompt> export MYVAR

Variables that are usually present are

  • HOME: your home directory, used when you give the command cd without arguments to get to your home directory
  • PATH: the list of directories searched for the name of the command you give without a / in it.<br /> NB Do not put the current directory “.” at the front of the path!
  • EDITOR: The editor command used when you invoke a command that uses an editor to let you modify files. Typical editor commands are vi, nano, joe
  • PAGER: The pager command used when e.g. a manual page doesn’t fit on a screen. Typical pagers are less and more. And many more…

What is input/output redirection

Whenever you start a command, the shell opens three “files” by default. These are stdin, stdout and stderr, also known by their numbers, 0, 1 and 2.

The (bourne) shell has the ability to redirect the flow of these files to somewhere else.

Without redirection, the three files are all connected to the terminal, so stdin to the keyboard device, stdout and stderr to the terminal window.

If you command the shell to redirect the output to a file, you can store the output of the command to a file, instead of seeing it on the window where you typed the command.

examples

ls > mylisting

output redirection creates or overwrites the file mylisting with the output of the ls command

ls >> mylisting

output redirection creates or appends the file mylisting with the output of the ls command

`ls some-file 2>myerrors`

standard error redirection creates or overwrites the file myerrors with the error message produced by the ls command

wc -l <$HOME/.bash_history

using input redirection count the number of lines in the file $HOME/.bash_history. Try the same command with and without the <. With the < the stdin is redirected from the file $HOME/.bash_history to the wc command. Without the < the filename is passed as an argument to wc so it can open the file itself.

Instead of redirecting to a file, it is also possible to redirect the output of a command to the stdin of another command. This is called a pipeline. Another way to look at this is to see the command after the | symbol as a filter. Multiple filters can be used in sequence to massage the output of a command into a desirable format or selection.

cat /etc/shells | wc -l

How can I run multiple commands from one commandline?

Normally, commands are separated by a newline <enter>, however, you can also use a ; semicolon to seperate commands:

ls; sleep 10; echo bla

The above will list the directory, wait 10 seconds, then print bla

How can I run a command non-interactively and without having to wait for it to finish?

A common problem is that you want to run a command that takes a long time to finish and can do so without needing any input.

You can start a command to run in the background by adding a &

ls -l &

As you will see when you try this, it is still connected to the terminal. If you prepend the command nohup you can disconnect it and even logout of the machine you run the command on. The output will typically end up in a file called nohup.out.

The shell has some commands that allow you to change the background, foreground and stopped state of a job (i.e. a running command).

  • The key control-Z stops a foreground job and returns you to the shell prompt
  • the command fg brings a stopped job or a background job to running in the foreground (access to the terminal)
  • the command bg brings a stopped command to the background
  • the command jobs lists the background and stopped jobs started from your shell
  • the command kill can send a signal to a background jobs. E.g. kill %1
    • NB, the kill command is most commonly used to send signals to process id’s (PID), which do not have to be started in the current shell.

Alternatively: screen

If you want to run the program and be able to provide input or check on its progress now and then, you can run a virtual terminal program, like screen or tmux, these will continue running when you disconnect and log out, later you can reconnect to see how it’s going. See man screen or man tmux.

What is filename expansion or wildcards?

The shell has some tools to save you typing long filenames. E.g. when you don’t want to list all files in a directory:

ls -d ? list all files and directories with a name of one character

ls -d ??? list all files and directories with a name of three character

ls -d a* list all files and directories with a name starting with “a” and any number of unspecified characters after it

ls -d .* list all files and directories with a name of three character

ls -d *[0-9] list all files and directories ending with a number

ls -d *[a-z] list all files and directories ending with a lowercase letter

ls -d *[A-Z0-9] list all files and directories ending with an uppercase letter or a number

ls -d ~/a* list all files and directories in my homedirectory starting with the letter a

ls -d *\** or list all files and directories with names containing a real ls -d *'*'* *. Use the \’ or single quotes to escape the character following it. i.e. prevent the shell from using the * for wildcard expansion.

characters to < > [ ] ( ) { } * & ! ~ ; “space” “quotes” and many escape more…

The option -d of ls makes sure to list the entry itself and not the contents of the directory by that name.

More information about wildcards can be found in the bash manual.

NB A special case is when you escape the newline or <enter> character. The shell will not start interpreting the command and you can add more to your command line on the next line.

ls \&lt;enter&gt;
-ltr&lt;enter&gt;
...
prompt&gt;

Will be the same as “ls -ltr”

How do I customise my interactive shell session?

In this section we assume you are using the Bourne Again SHell (bash). You can configure your login shell via DIY. (If you are running this on your own Linux machine, you can use the chsh command to change the login shell.)

What happens when I login?

When you login, bash checks a list of possible locations for configuration files (which are full of shell commands).

Bash looks for the following files

/etc/profile
~/.bash_profile
~/.bash_login
~/.profile.

When you start a new interactive shell on the same machine, bash looks for other files:

/etc/bash.bashrc
~/.bashrc

People often want to have the same settings in all shells, so a lot of times you see the following in ~/.profile:

if [ -f ~/.bashrc ]; then
        source ~/.bashrc
fi

And all configuration can exist in ~/.bashrc

In either the system or personal bashrc files, variables and aliases are set to preference:

  • PATH is set to find systemwide programs, you can expand this search path to include directories in your homedir
  • PS1 is set to a useful prompt
  • PAGER

How can I change the way my prompt looks?

The variable PS1 is the prompt you normally see when you can enter a command to the shell. Various ways exist to make this more useful.

  • Set PS1 directly
  • Set PROMPT_COMMAND to something that sets PS1

To make it permanent, define PS1 or PROMPT_COMMAND in your ~/.bashrc file.

examples

PS1='\u@\h:\w\$ '
PS1='\u@\h:\w [$?]\$ '
PS1='[\e[32m]\u@\h:[\e[33m]\w[\e[32m]$[\e[0m] '

Other customizations

Scripting

Often you want to be able to reproduce a sequence of commands or even create a small program to do a lot of work in the shell.

For this purpose, you can create a batch script, shell file, or shell script. We recommend you do this in the bourne shell (sh) or (bash). Using (t)csh for shell scripts is not recommended, see Csh Programming Considered Harmful.

Basic shell script

echo start of script
ls -l
echo end of script

If you put the above 3 lines in a file myscript, you can call bash myscript and run the commands in the file.

In order to make the file a command, you can make the file executable by giving the command: chmod a+x myscript

Then you can call the script using ./myscript

Choose interpreter #! hashbang

How to provide multiple lines of input: Here Documents

How to use the output of command as parameters: Command substitution

prompt> CMD-1 `CMD-2`
or ~equivalent in POSIX shell
prompt> CMD-1 $(CMD-2)

Call CMD-1 with arguments created from the output of CMD-2

example

Rename a file to have a different extension or a date in the name.

  • Get the date in a particular format (see man date)

    prompt> date “+%Y%m%d-%H:%M” 20180327-12:41

  • Get the first part of the name

    prompt> basename pietje.jpg .jpg pietje

  • Use both commands to create a new name for move (rename)

    prompt> mv -v pietje.jpg $(basename pietje.jpg .jpg)-$(date “+%Y%m%d-%H:%M”).jpg ‘pietje.jpg’ -> ‘pietje-20180327-12:45.jpg’

  • The CMD-2 will be executed before CMD-1.

  • Command substitution can be invoked using `CMD-2` or $(CMD-2)

  • The advantage of $() is that it can be nested $(CMD-2 $(CMD-3))

Unix Permissions

Linux and unix have read, write and execute/access permissions for the owner, the group and others. This concept works very well, but has some limitations.

One common mistake is that the permissions for “others” are often called “world”, which implies that the whole world has access to the file or directory. Others in this case means not the owner and not any member of the group.

Basic unix permissions

To find out about permissions on the filesystem, you can give the ls -l command, which shows the permissions in the first column:

prompt> ls -l
total 8.0K
drwx------ 2 simon simon 4.0K Mar 28 13:23 a
drwxrwxr-x 2 simon simon 4.0K Mar 28 13:23 b
-rw-rw-r-- 1 simon cncz     0 Mar 28 13:23 c
-r--r--r-- 1 simon simon    0 Mar 28 13:23 d
-rwxrwxrwx 1 simon simon    0 Mar 28 13:23 e

The output meaning is:

drwxr-xr-- 2 simon cncz  4.0K Mar 28 13:23 a
|||||||||| |   |     |     |        |      |
|||||||||| |   |     |     |        |      +- filename
|||||||||| |   |     |     |        +-------- modification date/time
|||||||||| |   |     |     +----------------- filesize
|||||||||| |   |     +----------------------- group
|||||||||| |   +----------------------------- owner
|||||||||| +--------------------------------- number of hardlinks to file
|||||||||+----------------------------------- execute bit for others
||||||||+------------------------------------ write bit for others
|||||||+------------------------------------- read bit for others
||||||+-------------------------------------- execute bit for group
|||||+--------------------------------------- write bit for group
||||+---------------------------------------- read bit for group
|||+----------------------------------------- execute bit for owner
||+------------------------------------------ write bit for owner
|+------------------------------------------- read bit for owner
+-------------------------------------------- directory (d), file (-) or ...

d/- directory (d) or file (-)

r r (permission to read) or - no permission to read

w w (permission to write) or - no permission to write

x x (permission to access/execute) or - no permission to access a directory or execute a file

The order of evaluation is like this:

  1. is the owner my effective user id? -> apply the permissions for owner -> finished
  2. am I a member of the group listed for this file/directory? -> apply the permission for the group -> finished
  3. am I not the owner and not a member of the group? -> apply the permissions for other -> finished

In case of directories, the execute bit has to apply for all the directories in the path all the way back to the root directory /.

Changing permissions

chmod

You can use chmod to change the permission bits.

The permissions are stored efficiently in the filesystem (in the inode). The permissions can be represented by 3 bits as an octal number ( 0 1 2 3 4 5 6 7 10 11 …)

The chmod command can use either a symbolic representation or octal representation of the permissions:

prompt> chmod u+rw,g=r,o= thefile

This is using chmod to change the permissions using a symbolic representation: change user permissions to add read and write permissions (don’t change execute permisions), set group to read and other to no permissions.

prompt> chmod 760 thefile

This is the octal way to represent the permissions, note that this cannot leave bits unchanged! You specify the permissions in three (or four) octal numbers, representing permissions for user, group and others. When using four numbers, this can set additional permissions, see advanced permissions.

octal binary permissions value value

0 000/— no permissions

1 001/–x only execute/access permission

2 010/-w- only write permssion

3 011/-wx write and execute permissions

4 100/r– only read permission (common for files)

5 101/r-x read and execute permissions (common for directory)

6 110/rw- read and write permissions (common for non executable files)

7 111/rwx read write and execute permissions (common for directories and executable files)

Important points

  • Note that directories need execute permissions to be usable (i.e. enter it)
  • Permissions can be changed recursively to all files/directories in the specified files/directories; chmod -R u+w somedir
  • changing permissions can be dangerous, you can loose access to your files or expose files to others that should not have access to them.

chown

Change the owner of a file/directory (only usable by root)

prompt> chown john somefile
  • chown also supports the -R flag

chgrp

Change the group of the file/directory

prompt> chgrp myothergroup somefile
  • chgrp also supports the -R flag

Advanced unix permissions

  • sticky bit

  • other types of files

  • umask

    3.2 Hoe kan ik mijn bestanden ontoegankelijk maken voor anderen?

    Daarvoor is zowel op Unix als op MSDOS met PC-NFS het ‘chmod’ commando. (Het bekijken van de protecties gaat met ’ls -lg’ ook op MSDOS!) Chmod heeft een aantal varianten om de ‘bitjes’ te zetten:

    H:> ls -lg bestandje -rwxr-xr-x groep eigenaar 112 bestandje

    ||| -> eigenaar ||| -> groep ||| -> wereld H:>

    Dus de protectie van directories en bestanden worden steeds door 3 ‘bitjes’ gegeven: rwx met de ‘waardes’ 4,2,1

    Het chmod commando wordt als volgt gebruikt:

    wn4> chmod u=rwx,g=rx,o= bestandje wn4> ls -lg bestandje -rwxr-x— groep eigenaar 112 bestandje

    Nu is het bestand alleen schrijfbaar voor de eigenaar, en lees en executeerbaar voor de eigenaar en de groep. De rest mag niets.

    Ander voorbeeld:

    H:> chmod g=,o= geheim H:> ls -lg geheim -rwx—— groep eigenaar 12 geheim

    Nu kan alleen de eigenaar het bestand lezen, wijzigen en executeren. Alle anderen kunnen niets.

    Ook kun je in chmod expliciet de bitjes opgeven, in octale notatie, zoiets als:

    wn4> chmod 700 bestandje wn4> ls -lg bestandje -rwx—— groep eigenaar 112 bestandje

    Hiermee is het bestand alleen nog te lezen, schrijven en executeren door de eigenaar.

    wn4> chmod 444 bestandje wn4> ls -lg bestandje -r–r–r– groep eigenaar 112 bestandje

    Nu is het bestand nog alleen nog te lezen door iedereen.

    Verder zie ‘man chmod’ en ‘man ls’ op Unix, voor PC-NFS staat e.e.a. in de manual.

    Bijbehorend is de umask variabele: deze maskeert bitjes die niet vanzelf aan moeten gaan voor nieuwe bestanden: Als de umask 022 is dan zullen de bitjes van de eigenaar niet beinvloed worden, het ‘2’ bitje (schrijven) voor de groep en de wereld wordt afgezet voor een nieuw bestand. Ook dit is weer waar voor zowel MSDOS als Unix. Onder MSDOS met PC-NFS zet men het umask als volgt:

    C:> net umask 077

    Dit zet alle toegang tot nieuwe bestanden voor anderen dan de eigenaar af. Onder Unix gaat het bijna identiek:

    wn4> umask 022

    Nu zijn voor alle nieuwe bestanden de ‘schrijf bitjes’ uit.

    Tenslotte: de protectie van de directorie is ook belangrijk! Het verwijderen van een file is een directorie operatie! Dus al heb ik op de file zelf geen rechten, als ik wel mag schrijven op de directorie kan ik het bestand gewoon verwijderen!

    3.3 Help, ik ben een file op Unix kwijt! Kan ik die terug krijgen?

    Zie http://www.sci.kun.nl/cncz/procedures/backups.html