Newniks: Tips for new Unix users

From xkcd.com

The term Unix is used here to refer to operating systems in the family of UNIX, OSF, GNU/Linux, FreeBSD, Mac OS X, etc.

The Art of Unix Programming by Eric Steven Raymond (2003) is a very interesting overall introduction to Unix.

This page is based on information that I started putting together for myself in 1991, as I started learning Unix (and csh) after years of using VMS. It started as a simple list of Unix equivalents for VMS commands but has grown beyond that. It is still intended mainly as an aide-mémoire for myself. For historical reasons, there may remain an indiscriminate mix of OSF and Linux (especially Debian GNU/Linux), and of the csh and bash shells.

Miscellaneous links (I don't actually make much use at all of these sites except for the Debian one):

• Debian GNU/Linux
• The Linux Documentation Project
• Linux.org, Linux Today
• Freecode (last updated in 2014)

The following sections are intended to be in order of increasing sophistication of the task. At the right is a table of contents with the section names sorted alphabetically. For Unix under MS Windows, see Cygwin & MinGW below.

1. Installation

These brief notes are for installing Debian GNU/Linux. See the Debian Web site for more details. These notes are no substitute for the real documentation.

1.1 Getting Debian

Debian CD and DVD .iso images can be downloaded from www.debian.org/CD/. Select one of the options for downloading.

If you select Download CD/DVD images using HTTP, read what the page says then scroll down to the list of registered mirrors and select one. Then select current, then your required architecture (most often amd64), then iso-cd, then select one of the .iso files, such as
debian-12.7.0-amd64-netinst.iso if you are using.

Once you have the .iso file, you may need to burn it to a CD or copy it to a USB stick. If you’re installing Linux under VirtualBox, you’ll just use the .iso file directly.

See Firmware for a discussion of how Debian handles non-free firmware, which may be required at installation time for some hardware (e.g., certain network drivers). Some Unofficial non-free images including firmware packages are available.

From xkcd.com

1.2 Installation procedure

For most of the questions asked during installation, either you can just accept the defaults or the appropriate answer is fairly obvious.

If you change your mind about your keyboard layout, you can change it later with the command
sudo dpkg-reconfigure keyboard-configuration
or by doing System ► Preferences ► Hardware ► Keyboard.

The host name may contain only the letters a to z, the digits 0 to 9, and the hyphen (-). You can change the host name later using the hostnamectl command.

When installing under VirtualBox, the best answers for host name and domain name are not obvious to me, but they may not matter very much under many circumstances.

1.2.1 sudo and root

When asked to enter a password for the ‘root’ user, it is better to just leave it blank. (In at least some Debian versions, if you specify a root password then the sudo command will not be installed, or at least you won’t be added to the list of users who can use it. Since it is better to use the sudo command than to become the root user when you need to do things that require special privileges, you do want to be able to use sudo.)

If you do specify a root password, you can later add yourself to the sudoers list.

If you don’t specify a root password here, the root account will be disabled. If you later do need a real root password (e.g., to get into recovery mode when booting) you can set one.

1.2.2 Disk setup

The default Guided - Use entire disk and All files in one partition are reasonable choices. The installer gives you several chances to change your mind about this choice, which is quite important. In most cases you can accept the default choices until the last step, when you must select Yes to actually do the disk setup.

When installing under VirtualBox, these choices are much less critical because it’s installing the guest on a virtual disk, not on the computer’s real disk. Don’t bother with logical volume management (LVM) unless you know you want it.

If you are installing Linux onto a computer that will also be running another operating system, things are more complicated and are not dealt with here; the wrong choices can lead to owning a very expensive paperweight. See Debian's Pre-Partitioning for Multi-Boot Systems, for example.

1.2.3 Mirror

Once the base system has been installed, you are asked whether you want to ‘scan more media’; if you just downloaded one .iso file, the answer is no. You are then asked to specify whether you want to use a ‘network mirror’ to ‘supplement the software that is included on the CD-ROM’. You should say yes. If you do not choose a mirror, only things that are on your CD or DVD (or in your .iso image) can be installed, some may be out of date, and you won’t get any security updates. For example, Firefox and GIMP are in amd64-xfce-CD-1.iso but LibreOffice and Inkscape are not, and none of them are in amd64-netinst.iso.

If you don’t specify a mirror and later you want to, it can be done by using a text editor (e.g., nano, which is in both amd64-xfce-CD-1.iso and amd64-netinst.iso) to add the mirror specification to the file /etc/apt/sources.list. The use of synaptic would be easier but it’s not in amd64-netinst.iso.

1.2.4 Desktop environment and tasks

If you specify a mirror, in the next step you can specify which graphical desktop environment you want (they’re very big but you probably want one) and which other tasks you want, in addition to the standard system utilities.

The installer will probably offer to install the Debian desktop environment and the standard system utilities by default, which you will presumably want to accept.

You’ll presumably also want to choose a specific desktop environment, several of which will be offered. I recommend the MATE desktop environment. It’s a continuation of GNOME 2, avoiding the many dubious design decisions that went into GNOME 3. (My software seems not to work well under GNOME 3, at least under VirtualBox, but works fine with MATE.) Xfce may be an interesting alternative that is similar to MATE in some ways (ref); I haven’t tried my software with it.

The installer will also offer to install a Web server and an SSH server. It’s up to you whether you want them. If you’re installing under VirtualBox, you probably don’t want them.

At this point you may be asked to put your CD in the drive; if you don't really have one (e.g., you installed in VirtualBox using a CD image) select Abort and it will download things entirely over the Internet from the mirror you specified earlier, without the CD. To avoid the request for a non-existent CD, you may need to edit the file /etc/apt/sources.list to remove (or comment out) the line corresponding to the CD.

The subsequent download and installation process may take a long time, depending on the speed of the network and the speed of your computer.

If necessary, you can subsequently install the MATE graphical desktop environment by doing
sudo apt install mate-desktop-environment

If you end up having multiple desktop environments installed, when you login you may need to select the one you want (e.g., MATE) by clicking on a Settings icon.

1.2.5 Installing a boot loader

You will generally want to install the GRUB boot loader to your primary drive. If offered a choice between 1: Enter device manually and something like 2: /dev/sda (…), you probably want to choose 2 unless you have special needs.

When installing under VirtualBox, you can go ahead and let the boot loader be installed without fear, because it’s just happening in your virtual disk.

1.2.6 Booting into the new system

If installing under VirtualBox, you may or may not need to remove the installation medium (i.e., the .iso file) before booting into the new system. Check Devices ► Optical Drives. If a .iso file has a check mark next to it, click on Remove disk from virtual drive.

If later you get messages telling you to insert the CD from which you installed the operating system when you use apt, aptitude or synaptic to install new software, it will be necessary to tell the system not to try to use a CD-ROM as a source: while running the guest Linux system, do System ► Administration ► Synaptic Package Manager then in Synaptic do Settings ► Repositories and uncheck the boxes corresponding to any entries that say cdrom.

At the beginning of the boot process, there may be a delay followed by the somewhat worrying message
Gave up waiting for suspend/resume device
This is normal. It can be suppressed if desired (ref).

1.3 After installation

• If you have a Mac with a 4K display and have installed Linux in VirtualBox, the guest machine needs to opened in low-resolution mode.
  • In versions of Mac OS before Monterey (ref):
    • Navigate to the package contents of the VirtualBox app and find VirtualBoxVM under Contents/Resources
    • In Finder, right-click on get-info and click on Open in low resolution, then change to "Scale to 100%" under View ► Virtual screen 1
  • In Mac OS Monterey, because of Apple’s ‘improvements’, you need to use a text editor (e.g., nano) to edit an Info.plist file (ref). If you’re reasonably paranoid, you will first make a backup copy of the Info.plist file. To edit the file:
    • In a terminal window in your Linux guest machine, give the command
      sudo nano /Applications/VirtualBox.app/Contents/Resources/VirtualBoxVM.app/Contents/Info.plist
    • Change the line
      <key>NSHighResolutionCapable</key> <true/>
      to
      <key>NSHighResolutionCapable</key> <false/>
    • Type CtrlX to exit from nano and then Y to save the modified file (overwriting the original file), followed by Return to use the original file name.
    Shut down the guest machine and restart it for the change to take effect.
• If installing under VirtualBox, install Guest Additions and then set up sharing of files between guest and host.
• If installing on a desktop computer in BME, do network setup including conversion from DHCP to a fixed IP address.
• For access to Probe and probeShare in BME, give the command
  sudo apt install cifs-utils
• Add an icon for the MATE Terminal to the top panel: right-click on the top panel, do Add to Panel… ► Application Launcher… ► Forward ► System Tools ► MATE Terminal ► Add ► Close
• Create a bin subdirectory in your home directory: open a terminal window and give the command
  mkdir ~/bin
  
  Once the ~/bin/ directory exists, it is added to the PATH environment variable when you login. However, some display managers do not consider the opening of a terminal window as being a login. You can give the command source ~/.profile to get ~/bin/ added to the PATH, but this needs to be done again whenever you open a new terminal window. The MATE Terminal can be customized as shown here to insist that opening it be considered as a login.
• Customize the MATE Terminal so the ~/bin directory will automatically be added to your PATH (the list of directories to be searched when you give a command). Open a MATE terminal window and in the menu bar at the top select Edit ► Profile Preferences. In the Title and Command tab, check the box Run command as a login shell, then click on Close. You’ll need to open a new terminal window to see the effect of this.
  

The following are things that I personally do after installation of the standard MATE desktop environment (and ssh and Web servers if desired). You may not want or need to do any of these things.

• Open a terminal window by clicking on the icon that I’ve added to the top panel. In the terminal window, do Edit ► Profile Preferences ► Scrolling and set the Scrollback value to some large number (e.g., 10,000). Setting it to Unlimited might be a bit dangerous if things get out of control.
• In a terminal window give the command
  sudo apt install aptitude
  This is because I’m used to using aptitude for package management. Alternatively you can use synaptic, which comes as part of the initial installation, or just use apt install from the command line for installing packages. In MATE 1.16.2, synaptic is accessed by doing System ► Administration ► Synaptic Package Manager.
• sudo apt install gnome-system-tools for doing things like adding users (mate-system-tools no longer exists in Debian Stretch.)
• sudo apt install info locate rsync sshfs
• sudo apt install emacs xournal k4dirstat
• Do sudo nano -B /etc/apt/sources.list to add contrib non-free to supplement main. (-B causes nano to create a back-up file when the modified file is saved. A back-up can also be requested from within nano when you save the file.) Do sudo apt-get update afterward.
• sudo apt install ttf-mscorefonts-installer to get a number of Microsoft fonts (Arial, Times New Roman, etc.)
• sudo apt install unrar
• Set up my customizations for Emacs
  • In home directory, do
    rsync -avx username@host:.emacs .
rsync -avx username@host:.emacs.d/lisp .emacs.d
  • To also copy Emacs stuff to /root (for when using Emacs as superuser):
    sudo -i
rsync -avx username@host:.emacs .
rsync -avx username@host:.emacs.d/lisp .emacs.d
exit
• Set up things for my fun scripts:
  mkdir ~/fun
cd ~/fun
rsync -avx username@host:fun/{getfun.pl,fun.pl,Fun.pm} .
cd ~/bin
ln -s ~/fun/fun.pl fun
  
• Download additional utility scripts for use with the fun command:
  fun getfun
• Edit ~/.bashrc
  • Remove : and \w (current working directory) from the prompts in terminal windows. Possibly also remove \u (username) and @. They each appear twice, for the colour and non-colour prompts.
  • Comment out alias ls='ls --color=auto' since many of the colour combinations are very hard to read.
  • Add
    alias cp='cp -i'
alias mv='mv -i'
    to reduce risk of accidental file deletion by cp and mv
• For AudiLab software
  • fun getsw to download AudiLab software into ~/bin/. Alternatively, you could put the software somewhere else and create symbolic links in ~/bin/ to point to where it’s saved. For example, in ~/bin/ do
    ln -s ~/Downloads/AudiLab/fie
  • sudo apt install libncurses5
  • Add icons for AudiLab software to the top panel: right-click on the top panel, do Add to Panel… ► Custom Application Launcher; for Type select Application in Terminal; specify a Name (e.g. Fie) and a Command (e.g., /home/username/bin/fie) then click on OK.

    If you get an error message because xterm is not found when you click on the new icon, one solution is to select Application instead of Application in Terminal and explicitly invoke the desired terminal in Command (e.g., mate-terminal -e '/home/username/bin/fie').

    If you want to change the icon of something in the panel, right-click on the icon and select Properties. In the Launcher Properties window, click on the icon. This will open a directory listing. Browse to find an icon that you like; .svg and 48×48 .png files are accepted, and maybe others.)

  • For compiling AudiLab software, install packages gfortran (and/or g95?), libncurses5-dev, xorg-dev (may be overkill), libjpeg-dev, libglib2.0-dev, libgtk2.0-dev (and the packages they depend on).
• For compiling Alpine, install packages libncurses5-dev (as in previous item), libpam0g-dev and libssl-dev (and the packages they depend on).
• In the file manager, select Edit ► Preferences and in the View tab set View new folders using to List View and check the boxes Show hidden files and Show backup files.
• To cause windows to always be grouped in the Window List (taskbar) at the bottom of the screen, right click on the handle () near the left end of the Window List and select Preferences. Under Window Grouping select Always group windows.

1.3.1 Notes on desktop when upgrading to Wheezy (Debian 7)

I started writing here about how I customize settings in GNOME 3 but then discovered that I couldn't. GNOME 3 and its default Adwaita theme are abominations. One could perhaps switch back to GNOME 2, or try MATE (a fork of GNOME 2) or Xfce.

In the meantime, make sure that the gnome-session-fallback package is installed, then when logging in select GNOME Classic rather than GNOME or System Default.

Also, install the clearlooks-phenix-theme package. It is ‘a GTK3 theme which is a port of Clearlooks, the default theme for GNOME 2’. Once installed, go to Applications ► System Tools ► Preferences ► Advanced Settings (or Tweak Tool with package gnome-tweak-tool installed) and select Clearlooks-Phenix for the GTK+ theme and the Window theme.

The gnome-tweak-tool package is supposed to allow ‘the adjustment of several advanced GNOME options that can’t be configured in gnome-control-center’ (see live.gnome.org/GnomeTweakTool). I haven't got it to work yet, it just spews out error messages then brings up the standard Advanced Settings dialogue.

Changing the date format to year-month-day should be easy but isn't in GNOME 3. See Chris Collins' post for one approach.

The command dconf-editor may provide some additional possibilities for customization.

To add a programme to the launch panel at the top of the screen do Alt RightClick on the panel.

To change whether windows are grouped or not in the taskbar at the bottom of the screen, RightClick at the extreme left of the taskbar, up against the edge of the screen, which gives the possibility of bringing up a Preferences window.

If the GNOME settings get hopelessly mangled, you may get a ‘Something has gone wrong’ message with no alternative but to log out, log back in and get the same message again. (This happened to me when I tried to move the bottom GNOME panel to the left side of the screen.) dconf-editor in the dconf-tools package is a GUI tool for editing the configuration of GNOME but it won't be usable if your GNOME GUI isn't usable. User configuration settings are in various places, including ~/.gconf/ and ~/.config/dconf/user. I recovered a working configuration by doing Ctrl-Alt-F1 to get to a text console and replacing the latter file with a backup version (called user.something) that was in the same directory. I had also deleted ~/.gconf/ and everything in it, but that in itself didn't help.

1.3.2 Notes on desktop when upgrading to Jessie (Debian 8)

In Jessie, GNOME 3 is more offensive than before, with the GNOME Classic very much changed. I have installed MATE (package mate-desktop-environment), the continuation of GNOME 2.

2. Software packages

2.1 Searching for packages

• Debian: Search package directories
• Ubuntu: Ubuntu Packages Search (scroll past the browsing section)

2.2 Package-management tools

This is the beginning of trying to understand the various package-management tools for installing and removing software that Debian provides.

See Raphaël Hertzog’s Pick the right Debian package manager for you.

* indicates that the package is included in the amd64-xfce-CD-1 CD, ** indicates that it also appears in the amd64-netinst CD, as of 2020 Jun 13.

• dpkg**: a command-line tool to ‘install, build, remove and manage Debian packages’. Not usually used directly by the user: ‘The primary and more user-friendly front-end for dpkg is aptitude’ (man page).
• apt**: a high-level command-line interface that is intended to be a better end-user interface than more specialixed tools like apt-get and apt-cache (man page). Depends on libapt.
• aptitude: text-based menus; depends on libapt, libept;
  Use sudo aptitude to invoke aptitude. Basic usage: / to search for a package name, n to repeat search, + to specify that a package is to be installed, g to get packages (i.e., do the installation), q to quit.
• synaptic*: GUI; depends on software-properties-gtk, libapt, libept
• dselect: text-based menus; depends on dpkg
  This is an old interface to dpkg, from before aptitude.
• software-center: GUI, depended on synaptic
• wajig: ‘A command-line wrapper around apt, apt-cache, dpkg, aptitude, and more. It's goal is to ease package management by unifying the main functions of these tools from one interface.’ The documentation and behaviour seem inconsistent (e.g., the existence of a -t option or not) and it may be defunct (e.g., commands refer to apt-get.org, which was last crawled by archive.org in 2014).
• python-software-properties, software-properties-gtk, software-properties-kde: for managing sources; depend on python-apt which depends on libapt
• packagekit: 'simple'; depends on package manager of a particular distribution (libapt for Debian)
• packagekit-tools: command line, depends on packagekit
• gnome-packagekit: GUI, depends on packagekit; includes gpk-application (Add/Remove Software), gpk-update-viewer (checks for updates), gpk-prefs (settings for updates, including when to check, what to install, and which repositories to use), gpk-repo (enabling or disabling software repositories; now in gpk-prefs; doesn't allow adding contrib and non-free, doesn't allow specification of a mirror), gpk-log (log viewer), gpk-install-*, etc.;
• gnome-software: ‘Software’ ‘lets you install and update applications and system extensions … a PackageKit plugin provides data from a number of traditional packaging systems, such as rpm or apt. An appdata plugin provides additional metadata from locally installed data in the appdata format.’

Applications ► System Tools ► Add/Remove Software invokes gpk-application; available when Debian desktop (GNOME) is installed from CD without mirror.

The Debian CD search engine can be used to find out which CD a particular package is in.

2.3 Sources

The sources (repositories) from which Debian package management obtains packages are listed in the file /etc/apt/sources.list and possibly in similar files in the /etc/apt/sources.list.d/ directory. As of Debian 9 (stretch) these files can use either multiline ‘stanzas’ or the older one-line style.

The list(s) of sources can be created and/or modified in various ways:

• Use a special interactive utility such as synaptic.
• Use a text editor. The nano text editor is on CD 1.
• Use tee (or possibly > or >>) from the command line.

2.4 Mirrors

Tools to help in selecting a ‘mirror’ (i.e., a Web server that provides a replica of the main Debian repository) that is relatively close and fast.

• netselect: tests speeds of specified mirrors (on amd64 CD 11)
• netselect-apt: tests speed of all Debian mirrors and creates a sources.list file; depends on netselect (on amd64 CD 13) For example, sudo netselect-apt creates sources.list in current directory.
• apt-spy: tests speed of Debian mirrors in a specified region and creates a sources.list file. (on amd64 CD 13) For example, sudo apt-spy -s ca -d stable for Canada; creates /etc/apt/sources.list.d/apt-spy.list, which will be used unless deleted or renamed not to end with .list.

I haven't yet figured out how to add or change a mirror, based on what is available on CD 1, without using a text editor. apt-mirror-setup is what is used during installation but it not supposed to be (cannot be?) used outside the installer.

2.5 Cleaning up old packages

See Raphaël Hertzog’s posts on Debian Cleanup:

• 1. Get rid of useless configuration files
• 2. Get rid of obsolete packages
• 3. Get rid of third-party packages
• 4. Find broken packages and reinstall them
• 5. Identify cruft that can be removed from your Debian system
• 6. Remove automatically installed packages that are no longer needed

2.6 .deb file format

Debian packages are stored in .deb files, which are normally handled using one of Debian’s package-management tools. A .deb file consists of an ar archive that in turn contains compressed tar archives. If desired, the contents can be extracted by doing
ar x name.deb
tar xvf data.tar.xz
tar xvf control.tar.xz [scripts and meta-information]
more debian-binary [package-format version number]

3. Command-line editing

The details of command-line editing depend on which shell you’re using. For bash, see Command Line Editing in the GNU Bash Manual.

4. Wildcards and patterns

The pattern matching implemented in shells has some similarities to, but is not the same as, the use of regular expressions.

4.1 Pattern matching in the bash shell

Simple wildcards:

• * matches any string (including the null string)
• ? matches any single character
• [...] matches any one of the enclosed characters, with special meaning for - (range) and ! or ^ (not), and other fancy things

Use the built-in bash command shopt -s extglob to turn on extended patterns. Then, for example, ls *.*(cpp|h|ui) will list all .cpp, .h and .ui files.

4.2 Regular expressions

grep uses regular expressions. By default it understands basic regular expressions. To use extended regular expressions, use grep -E or egrep.

For example, to find lines of the file test.txt that contain either ‘aaa’ or ‘bbb’, use the command grep -E 'aaa|bbb' test.txt

To find lines of the file test.txt that do not contain either ‘aaa’ or ‘bbb’, use the command grep -vE 'aaa|bbb' test.txt

To find lines that contain ‘aaa’ that is not immediately preceded or followed by a non-alphanumeric character (which is useful for removing most of the extraneous hits in an e-mail file that contains encoded binary files), use grep -E [^[:alnum:]]aaa[^[:alnum:]] test.txt

‘In GNU grep, basic and extended regular expressions are merely different notations for the same pattern-matching functionality. In other implementations, basic regular expressions are ordinarily less powerful than extended, though occasionally it is the other way around’ (grep).

5. File management

5.1 Making and deleting directories and files

If using the command line rather than a GUI:

• mkdir to make a new directory within the current directory
• rmdir to delete (remove) a directory
• rm to delete (remove) a file

5.2 Directory listings

• ls for command-line directory listings. Its sorting is controlled by LC_COLLATE; normally it is case sensitive, and sorts the numeric parts of file names as ordinary character strings.
• Nautilus is the GNOME file manager. Its sorting is case insensitive, and it sorts the numeric parts of file names in numerical order rather than as character strings (e.g., x9.tmp comes before x10.tmp).
• Konqueror is the KDE file manager. Its sorting can be either case sensitive or case insensitive; it sorts the numeric parts of file names as ordinary character strings. It can also act as a media player, Web browser, FTP client, etc.

Each file has 3 timestamps associated with it:

• mtime: the last time the file was modified (including its creation)
• ctime: the last time the file’s ‘inode’ was changed (e.g., when the file itself or its owner or permissions were changed)
• atime: the last time the file was accessed

The ls command displays the mtime by default, but can be asked to list the atime (-u) or the ctime (-c) instead. If you want to display more than one of the timestamps at the same time, you can use stat or find; for example,
stat -c %x,%z,%n *
find ./ -printf "%Ab %Ad %AH:%AM\t%Cb %Cd %CH:%CM\t%P\n"

The touch command can be used to change the mtime to whatever you want (simultaneously changing the ctime to the current time). For example,
touch -t [[CC]YY]MMDDhhmm[.ss]

5.3 Finding files

• find searches for files within a specified directory tree, and can perform various operations on the files that are found. The command takes various options, tests (e.g., file size, time of last file access, filename pattern or regular expression) and actions (e.g., print file name, execute command). For example, to find Fortran source files and search for a text string in each one,
  find ./ -name \*.for -exec grep -H string {} \;
  To find PDF files and search for a text string in each one,
  find ./ -name \*.pdf -exec pdftotext {} - \; 2> /dev/null | grep -i string
  For the time-related options (-mtime, -ctime and -atime for days, or -mmin, -cmin and -amin for minutes), prefix the numeric argument by - to list more recent files and by + to list older files.
• locate works from a database that is periodically updated, so it's faster than find. The database is updated by the programme updatedb, which may be run manually or, for example, by the find script in /etc/cron.daily/. If the given pattern contains a wildcard, then it should probably start with a wildcard to allow for the unknown directory path (e.g., locate *libglib*). locate is in the package of the same name.

5.4 Disk usage

If your disk has filled up or is getting full, there are multiple ways of seeing how much space is left and of finding out what is using up the space.

5.4.1 Command line

• du: report disk usage. Common options:
  • -h for ‘human-readable’ format
  • -x to skip file systems other than the current one (e.g., an NFS mount)
• df: report disk free space. Common options:
  • -h for ‘human-readable’ format
  • -x fstype to exclude file systems of type fstype (e.g., vboxsf, nfs)

5.4.2 GUI

See my Swil.

6. Privileges

When superuser privileges are required, use sudo rather than logging in as the root user (e.g., using the su command), to avoid accidents.

A user can be given the sudo privilege by adding their username to the file /etc/sudoers or by adding them to the sudo group with the command
usermod -a -G sudo username
This must be done as superuser (i.e., after giving the su command and the root password). The user must log out and back in again for this to take effect.

When using sudo you can't access built-in shell commands like cd (ref). This can be worked around by doing sudo bash, sudo -i or sudo su. You should exit from the resulting privileged shell as soon as possible.

Note that su stands for ‘subsitute user’ and not ‘superuser’, since the command can be used for temporarily becoming any user, not just root.

If you need a real root password (e.g., to get into recovery mode when booting) and have not set one or do not remember it, you can use the command sudo passwd root to set it. If for some reason you cannot get logged in to use the sudo command, you can follow these instructions (e.g., ref):

• Start the boot process. When it gets to the screen where GRUB offers you a choice of how to boot, use the arrow keys to select recovery mode then press e (not Enter).
• In the sequence of boot commands that is displayed, use Ctrl-N or the arrow keys to go to the line that starts with linux.
• Use Ctrl-E or the arrow keys to position yourself at the end of the line. Add a space character to the end of the line if there is not one already.
• Add init=/bin/bash to the end of the line.
• Use Ctrl-X to continue the boot process. You should end up at a shell prompt where you can enter commands as root without having to give a password.
• Give the command mount -rw -o remount / to give yourself write access to the file system.
• Give the command passwd and specify a root password. You should see a message saying password updated successfully.
• Reboot.

7. Background processing

7.1 Running after logout

The '&' at the end of a command provides background processing that continues after logout. The 'nohup' command can be used to ignore hangups, but is necessary only to ignore explicitly sent hangup signals, since the '&' effectively protects a process from being sent a hangup signal upon logout.

7.2 Reduced priority

The 'nice' command (a different 'nice' command is built into csh) can be used. However, it may not be necessary: in 'UNIX for VMS Users' (Bourne, 1990) it says that 'in most versions of Unix the shell automatically lowers the priority of processes running in background for a system defined period of time' (p. 818), although I haven't found further details about this. The 'renice' command can be used to lower the priority of an already-running job. Note the stupid convention that the 'priority' parameter increases with decreasing priority.

7.3 Scheduling

The at command provides scheduled running of a process. The related batch command runs jobs when the average system load level falls below some specified value. The jobs are run using the sh shell.

cron is a dæmon for executing scheduled commands. It searches its spool area (/var/spool/cron/crontabs) for crontab files, which are named after user accounts, and runs them; these crontab files should be set up using the crontab command. cron also processes the file /etc/crontab, which in turn invokes any files in /etc/cron.hourly/, /etc/cron.daily/, /etc/cron.weekly/ and /etc/cron.monthly/. (One can also put crontab files in /etc/cron.d/ but this is deprecated.)

7.4 Notification

Shell provides notification when background process terminates. Normally notification is given just before the system prompt reappears. The shell variable 'notify' (or the 'notify' command) will cause notification to be immediate.

7.5 Queue management and distributed processing

Torque (Debian packages torque-*) and Grid Engine provide facilities for queue management on a single machine or more generally across distributed machines. The Grid Engine project, formerly developed by Sun, has now split into a commercial project (Univa) and two competing open-source projects, Open Grid Scheduler and Son of Grid Engine. Before the split, Debian included the packages gridengine-* for the Sun product, but as of 2015 May 9 the situation in Debian is unclear to me.

gridengine.org is said to be a useful resource.

Globus (Debian packages globus-*) provides facilities and tools to help researchers ‘move, manage, and share big data’.

8. Linking

Linking is the process of linking a compiled programme together with its libraries. It may also be referred to as building, and in Unix it is done by ld, the loader. In the case of shared libraries, the linking must take place both at link (build) time and at run (load) time.

The linker ld has a built-in search path for finding required libraries. The default search path seems to include only /lib/ and /usr/lib/. A directory can be added to the default search path by adding it to /etc/ld.so.conf and then running ldconfig. The command ldconfig -v can be used to see what libraries ld is aware of.

One can also specify additional search directories on the ld command line using the -L and -R flags.

There is also a mechanism involving the environment variables LD_LIBRARY_PATH (deprecated except for temporary kluges) and LD_RUN_PATH. See Russ Allbery's Shared library search paths for a good discussion.

The purposes of the various directories like /lib/ and /usr/lib/ are specified in the Filesystem Hierarchy Standard.

9. Debugging

The following are commonly used commands within dbx:

run	Run from beginning. Command-line arguments can be included as they would be when invoking the programme from the shell. Redirection of stdin and stdout can also be included in the usual way.
next (n)	Execute one line (do not enter procedures)
step (s)	Execute one line (do enter procedures)
stop in proc	Stop when procedure is called (set breakpoint)
stop at line	Stop when given line in current procedure is reached (set breakpoint)
delete all	Delete (disable) all breakpoints, trace events and record events
cont (c)	Continue
return	Execute until return from procedure
list (l) n1,nl	Lists lines in the current source file
print (p) expr	Print value of expression
where (t)	Display list of active procedures
use dir	Sets list of directories searched for source files (~ not supported, so must specify, e.g., /usr/users/name/src rather than ~name/src; separate paths by spaces)
quit (q)

10. Make

Running make recompiles only routines modified since the last make. In order to recompile everything, must use touch to fake the modification dates(!).

11. Environment & shell variables

The shell variables user, term, home & path are automatically mirrored in the environment variables USER, TERM, HOME & PATH.

Environment variables may be def'd in .login or .cshrc

12. Aliases

Aliases may be def'd in .login or .cshrc

The mv and cp commands by default will overwrite existing files without warning. It is prudent to define aliases which invoke the -i flag, which causes a prompt before overwriting. With csh under Tru64 Unix, the -i in the alias can be overridden just by giving the command cp -f. With tcsh under Debian Linux, this doesn't seem to work, but the alias substitution can be avoided by giving the command \cp.

Note a nasty way of messing up using the cp command: if you say, e.g., cp *.dcl instead of cp *.dcl ., and if there are exactly 2 .dcl files, it will copy the first over the second without warning that the syntax is wrong.

13. Fonts

This sample set of font specifiers includes all combinations of serif or sans serif, medium or bold, upright (roman) or oblique (italic), and proportionally spaced or monospaced (fixed pitch):

For a serif font, italic is indicated by -i-; for a sans-serif font, oblique is indicated by -o-.

The DejaVu fonts are an extension of the very high-quality Vera fonts.

Use xfontsel, xlsfonts or xfd (all in package x11-utils) to explore available fonts.

Fonts in Debian (2017)

14. Tar

Example of using tar to transfer files from VMS:

    On VMS:
      $ FUN TAR
      $ FUN TO GUT
      $ VMSTAR_U cvf GUT_FOR.TAR *.FOR
    On OSF:                     On VMS:
      % cd gut
      % ftp funvie                $ FTP PALACE
      ftp> cd fun_gut             FTP>SET DEF "gut"
      ftp> image                  FTP>SET TYPE IMAGE
      ftp> get GUT_FOR.TAR        FTP>PUT GUT_FOR.TAR
      ftp> quit                   FTP>EXIT
      % tar xvf GUT_FOR.TAR 

15. Scripts

A script is invoked by typing its name, like any other command.

A csh script must start with a comment (#) in order to be recognized when invoked directly. (It's unnecessary if it's invoked with a csh or source command.)

Script files must be made executable, e.g. chmod u+x name.

16. Shells

16.1 Alternatives

To find out which shell is being used, look at the $SHELL environment variable (e.g., echo $SHELL).

Use chsh to change the default shell for a user (e.g., chsh -s /bin/bash).

The following are alternative Unix shells. (References: John Shepherd's notes from 2004; Sven Mascheck's notes; Wikipedia article.)

Bourne-shell family
sh	Steve Bourne	early 70's	the original Unix shell	uses .profile
Rsh			Restricted version of Bourne shell
ksh	David Korn	early 80's	like sh + functions, history editing, ...
ash	Kenneth Almquist	late 80's	small and fast version of sh
zsh	Paul Falstad	late 80's	like ksh but many, many enhancements
bash	Ramey/Fox	early 90's	like ksh with extras; default Debian interactive shell	uses .bashrc & .bash_profile
dash	Herbert Xu, Debian	late 90's	POSIX; default Debian /bin/sh shell
C-shell family
csh	Bill Joy	late 70's	original C-syntax shell, + job control, history	uses .cshrc & .login
tcsh	Ken Greer	early 80's	like csh + extra interactive features
rc	Byron Rakitzis	early 90's	even more C-like syntax, from Plan-9

16.2 Login scripts

The following is a simplification.

16.2.1 sh (Bourne shell)

sh executes ~/.profile when started.

16.2.2 csh (C shell)

csh executes ~/.cshrc when started, then executes ~/.login if invoked as a login shell.

16.2.3 bash (Bourne-again shell)

As a login interactive shell, bash executes ~/.bash_profile, ~/.bash_login or ~/.profile.
As a non-login interactive shell, bash executes ~/.bashrc (which is typically also executed early by ~/.bash_profile). (Environment variables that are exported in .bash_profile will be inherited by a subsequent non-login shell; aliases are not inherited and so are typically defined in .bashrc.)
As a non-interactive shell, by default no files are executed.

The GNOME terminal-window application, by default, does not run as a login shell, so it inherits things from the initial GNOME login. This means, for example, that, if ~/.bash_profile is modified, the changes won't take effect if you just invoke a new terminal window; you'll need to actually log out and back in to your GNOME session. Similarly, if you initially don't have a ~/bin/ directory and then create one, you'll have to log out and back in for it to show up in your PATH. This behaviour can be changed by checking the Run command as a login shell box under the Command tab, accessed by doing Edit ► Preferences ► Profiles and Edit or by doing Edit ► Profile Preferences.

17. Network setup

To set up a fixed IP address for an Ethernet connection, use System ► Preferences > Internet and Network > Network Connections. (If this item doesn't appear in the menu, make sure the package network-manager-gnome is installed.) In the Wired tab, select an existing connection and click Edit, or click Add. In the box that pops up, go to the IPv4 Settings tab. Under Method select Manual. Enter the IP address that has been assigned (e.g., 132.206.111.nnn). Enter the netmask (e.g., 255.255.255.0), the gateway address (e.g., 132.206.111.1, the IP addresses of some DNS servers, and the name of a domain to be searched (e.g., campus.mcgill.ca).

An alternative to the above GUI method is to edit the configuration file /etc/network/interfaces (ref).

Add the line ALL: ALL to the file /etc/hosts.deny and set up /etc/hosts.allow appropriately (e.g., with a line ALL: .mcgill.ca). See the Securing Debian Manual for details about security.

Logins from machine to machine should be done using ssh (in the ssh-client package). If you can't connect to a machine, make sure the ssh-server package is installed on that machine.

When you connect to a remote computer the first time, you'll be asked to confirm its ‘RSA host key’. If its key subsequently changes (e.g., when its operating system is reinstalled) you'll get an urgent-sounding warning. One way to deal with it is to give the commands
ssh-keygen -R hostIP
ssh-keygen -R hostname
where hostIP is the IP address of the remote computer (e.g., 123.456.789.123) and hostname is its name (e.g., name.dept.mcgill.ca). This removes the key from ~/.ssh/known_hosts, and then you'll be asked to confirm the new one when you try again to connect.

18. Telnet & FTP

telnet:
  Use  set escape '^@'  to change escape character.
ftp:
  Transferring multiple files:
    In Unix:
      Use mget & mput; invoke ftp with -i to disable prompting.
      Use ;0 with mget from VMS if only latest versions are desired
        (version numbers will appear in Unix filenames as ;n).
      Attempting to mget/mput large numbers of files will cause
        errors creating data sockets.
    In VMS:
      Use GET/PUT with wildcards; /NOCONFIRM is default.
      Use ;0 with mput if only latest versions are desired
        (version numbers will appear in Unix filenames as .n).
      Large numbers of files can be transferred without errors
        (it pauses occasionally, perhaps to clean up sockets).

19. System administration

19.1 Mail

To change limit on message size, edit MaxMessageSize= in /var/adm/sendmail/sendmail.cf . (Funsan receives mail but puts it into INBOX's on Cortex, so the message size limit depends on the limit set on Cortex.)

19.2 System logs

System logs are stored in or under /var/log/. User-space (non-kernel) boot-time messages are not logged by default; to enable them, install package bootlogd.

19.3 Web server

Debian has its own way of configuring the Apache Web server, as described in /etc/apache2/apache2.conf. It uses its own environment variables, defined in /etc/apache2/envvars; they are not displayed by the set command.

CGI. CGI scripts are usually handled by the cgid module, although there is also a cgi module. The default configuration file cgid.conf contains just
ScriptSock ${APACHE_RUN_DIR}/socks/cgisock
The default value of APACHE_RUN_DIR is /var/run/apache2, and the socket used for communication with the CGI daemon is /var/run/apache2/socks/cgisock.pid (with a file type of s).

19.4 Finding out what's going on

• lsb_release -cs, lsb_release -rs - display code name or release number of installed distribution in short form
• cat /etc/debian_version - shows which version of Debian is running
• uname -a - shows system information: hostname, hardware platform, kernel version, etc.
• hostname - shows (or sets) the system host name; options include showing domain name and DNS domain name
• more /etc/resolv.conf shows which DNS nameservers are being used (among other things)
• echo $DESKTOP_SESSION shows which desktop environment is in use, maybe (ref)
• nproc - shows number of processing units available
• fuser - shows which processes are using specified file(s) or file system
• iostat - reports I/O statistics
• lsof - lists open files
• ps - lists active processes (e.g., ps auxw)
• top - for CPU and memory usage, etc. By default, displays a selection of fields for running processes every few seconds, but is very customizable. For example, type f (or F) while it’s running to change the fields than are displayed, and type W to write out a configuration file with the current settings. Memory fields for processes include %MEM (share of physical memory), RES (amount of physical memory), SWAP (amount of non-resident memory), VIRT (= RES + SWAP) and others.
• iotop - for disk i/o, etc. (package iotop, run with sudo)
• iostat - for disk i/o per device, etc. (part of sysstat package)
• apt-file search filename - find out which packages contain the specified file; first use apt-file update to update the lists of package contents. apt-file is in the package of the same name.
• ar t - list the files contained in archive (library) files
• nm - list the symbols from object files, including libraries
• errno (from the moreutils package) - list or look up the error names and descriptions corresponding to errno values

19.5 Start up

The /sbin/init programme initializes the system by creating and controlling processes.

On Funsan, the system is considered to run at one of four run levels:

• 0: a safe state to turn off power
• s: single-user
• 2: multi-user local
• 3: multi-user networked

For each run level n, a directory rcn.d/ contains links which point to scripts in init.d/. On Funsan the rcn.d/ and init.d/ directories are in sbin, on Fundus they're in /etc/.

On Funsan, there are scripts /sbin/rcn for bringing the system to level n. I don't know how (or if) they're used, or if they also exist on Fundus.

19.6 Login

When users log in, the welcome message is taken from /etc/motd ('message of the day').

20. Cygwin & MinGW

Cygwin and MinGW provide environments for running *n*x under Microsoft Windows. See Software I like: Cygwin and MinGW.

21. X and GUI’s

The X Window System (often called X11) is a client-server software system that provides low-level functions from which to build graphical user interfaces (GUI’s) and applications for either networked or local computers.

In contrast to the usual usage of local clients and remote servers, the X11 client is the computer (possibly remote) on which the graphical application is running, and the server is the local computer on which the display occurs.

To run a programme remotely and have its display appear on your local computer, just use the -X option with your ssh command when logging in to the remote computer. (This assumes that both computers are running Unix.)

The Unix GUI consists of

• a desktop environment
• a window manager which can be tiling, stacking or compositing (i.e., compositing a set of off-screen buffers, one for each window)
• a session manager which saves and restores sessions
• a display manager which handles logins graphically

The MATE desktop environment is a continuation of GNOME 2, for those (like me) who don't like the direction GNOME 3 went in to copy Ubuntu’s Unity desktop environment and become ‘modern’ (and more like Mac OS X?). Unity is not available in Debian, and Ubuntu 18 will be switching back to GNOME (ref). KDE's desktop environment is Plasma.

Metacity is a light-weight window manager and was the default window manager for GNOME 2; Marco is its continuation under MATE. As of 2017 Jun, Compiz is already available as an alternative to Metacity in Ubuntu and will be in Debian 9 (stretch) when it comes out soon. Compiz makes use of OpenGL for fancy (gimmicky?) 3-D effects.

Matt Chapman’s guide to window managers for X seems to have stopped being updated before the introduction of the abomination that is GNOME 3, and thus before MATE.

Wayland ‘is intended as a simpler replacement for X’ and ‘GNOME and KDE are expected to be ported to it’.

The original low-level method of accessing the X protocol for client-side programming is Xlib (libX11). A newer method is XCB (libxcb), which was started in 2001. Since 2010, libX11 has only been implemented on top of XCB, and calls to both can be mixed (ref).

22. Backups

Resources for Amanda:

• Using Amanda (book chapter by John Jackson)
• FAQ

23. VMS/Unix equivalents

VMS commands for which Unix equivalents are given:
copy, create, define, del, diff, dir, dump, help, inquire, lo, rename, search, set, show, submit, type

23.1 copy

23.2 create

23.3 define

set or setenv

23.4 del

23.5 diff

23.6 dir

Fields in -l directory listing: type, protection, no. links, owner, group, size, date, name.

Date shown is date modified; -lu for date used, -lc for date created.

23.7 dump

23.8 help

Source: xkcd.com

23.9 inquire

23.10 lo

23.11 rename

23.12 search

23.13 set

23.14 show

23.15 submit

Facilities for background processing in Unix are quite different from the VMS submit command.

23.16 type