Qi user guide

Table of Contents

This user guide is for Qi (version 1.3, 10 Sep 2019).

Copyright (C) 2019 Matias Fonzo.

Qi’s home page can be found at http://www.dragora.org. Send bug reports or suggestions to dragora-users@nongnu.org.

1 Introduction

Qi is a simple but well-integrated package manager. It can create, install, remove, and upgrade software packages. Qi produces binary packages using recipes, which are files containing specific instructions to build each package from source. Qi can manage multiple packages under a single directory hierarchy. This method allows to maintain a set of packages and multiple versions of them. This means that Qi could be used as the main package manager or complement the existing one.

Qi offers a friendly command line interface, a global configuration file, a simple recipe layout to deploy software packages; also works with binary packages in parallel, speeding up installations and packages in production. The format used for packages is a simplified but safe POSIX pax archive compressed with lzip.

Qi is a modern (POSIX-compliant) shell script released under the terms of the GNU General Public License. There are only two major dependencies for the magic: graft(1) and tarlz(1), the rest is expected to be found in any Unix-like system.

2 Invoking qi

This chapter describes the synopsis and command line options for invoke Qi.

Usage: qi [OPTION]... [FILE]...

One mandatory option specifies the operation that ‘qi’ should perform, other options are meant to detail how this operation should be performed.

qi supports the following options to operate:

-b

Build package using recipe names.

-c

Create .tlz package from directory.

-d

Delete packages.

-i

Install packages.

-o

Resolve build order through .order files.

-u

Update packages (implies -i, -d and -p options).

-w

Warn about files that will be linked.

-x

Extract a package for debugging purposes.

There are common options between modes:

-N

Do not read the configuration file.

This will ignore any value in the qirc file.

-P <DIR>

Package directory for installations.

This option sets ‘${packagedir}’.

Only valid for -i, -d, or -u options.

-f

Force option.

This option can force the build of a recipe, or force the update of a pre-existing package.

Only valid for -b, -u options.

-t <DIR>

Target directory for symbolic links.

This option sets ‘${targetdir}’.

Only valid for -i, -d, or -u options.

-k

Keep (don’t delete) ‘${srcdir}’ or ‘${destdir}’ in build mode, keep (don’t delete) package directory in delete mode.

Only valid for -b, -d or -u options.

-p

Prune conflicts on package installations.

This option may proceed with the package installation if one or more conflicts occur.

-r /rootdir

Use the fully qualified named directory as the root directory for all qi operations. The target directory and package directory will be relative to the specified directory, including the log file for graft.

-v

Be verbose (a 2nd -v gives more).

Options for build mode (-b):

-O <DIR>

Where the packages produced are written.

This option sets ‘${outdir}’.

-W <DIR>

Where archives, patches, and recipes are expected.

This option sets ‘${worktree}’.

-Z <DIR>

Where (compressed) sources will be found.

This option sets ‘${tardir}’.

-a

Architecture to use.

Default value is obtained via uname(1) as ‘uname -m’.

-j

Parallel jobs for the compiler.

If not specified, default sets to 1.

-1

Increment release number (‘${release}’ + 1).

It will be omitted if the -n option is being used.

-n

Don’t create a .tlz package.

-S

Selects the option to skip completed recipes.

This means, in interactive mode, when the dialog to summarize recipes is shown.

Informative options:

-L

Print default directory locations.

This will print the target directory, package directory, working tree, the directory for tarballs, and the output directory for the packages produced.

-h

Display the help describing the options and then exit.

-V

Print the version number and license information. The version number should be included in all bug reports.

Expected non-option arguments are package directories and regular files: recipes or files ending in .tlz, .order. When FILE is -, qi can read from the standard input. See examples in Packages.

3 The qirc file

The global qirc file offers a way to define variables and tools (such as a download manager) for default use. This file is used by qi at runtime, e.g., to build, install, remove or upgrade packages.

It has the following rules:

• Variables must be declared as ‘name=value’.
• Declaration of values should only take one line, no line break.
• For security reasons, assignments like ‘name=$var’ are only interpreted as literal.

The command line options related to the package directory and target directory plus some of the options used for the build mode can override some values in qirc. See Invoking qi.

The order in which qi looks for this file is:

1. ${HOME}/.qirc Effective user.
2. ‘${sysconfdir}/qirc’ System-wide.

If you intend to run qi as effective user, the file ‘${sysconfdir}/qirc’ could be copied to ${HOME}/.qirc setting the paths for ‘${packagedir}’ and ‘${targetdir}’ according to the $HOME.

4 Packages

A package is a suite of programs usually distributed in binary form which may also contain manual pages, documentation, or any other file associated to a specific software.

The package format used by qi is a simplified POSIX pax archive compressed with lzip. The file extension for packages is ‘.tlz’.

Both package installation and package de-installation are managed using two important (internal) variables: ‘${packagedir}’ and ‘${targetdir}’, these values can be changed in the configuration file or via options.

‘${packagedir}’ is a common directory tree where the package contents will be decompressed (will reside).

‘${targetdir}’ is a target directory where the links will be made by graft(1) taking ‘${packagedir}/package_name’ into account.

Packages are installed in self-contained directory trees and symbolic links from a common area are made to the package files. This allows multiple versions of the same package to coexist on the same system.

4.1 Package conflicts

All the links to install or remove a package are handled by graft(1). Since multiple packages can be installed or removed at the same time, certain conflicts may arise between the packages.

graft¹ defines a CONFLICT as one of the following conditions:

• If the package object is a directory and the target object exists but is not a directory.
• If the package object is not a directory and the target object exists and is not a symbolic link.
• If the package object is not a directory and the target object exists and is a symbolic link to something other than the package object.

The default behavior of qi for an incoming package is to ABORT if a conflict arises. When a package is going to be deleted, qi tells to graft(1) to remove those parts that are not in conflict, leaving the links to the belonging package. This behavior can be forced if the -p option is given.

4.2 Installing packages

To install a single package, simply type:

qi -i coreutils-8.30-i586+1.tlz

To install multiple packages at once, type:

qi -i gcc-8.3.0-i586+1.tlz rafaela-2.2-i586+1.tlz ...

Warn about the files that will be linked:

qi -w bash-5.0-i586+1.tlz

This is to verify the content of a package before installing it.

See the process of an installation (very verbose):

qi -i -v mariana-3.0-i586+1.tlz

A second -v gives more.

Installing package in a different location:

qi -r /media/floppy -i lzip-1.21-i586+1.tlz

The -r option assumes ‘${targetdir}’ and ‘${packagedir}’. See:

qi -r /home/selk -P /pkgs -t / -i lzip-1.21-i586+1.tlz

In this case the content of "lzip-1.21-i586+1.tlz" will be decompressed into ‘/home/selk/pkgs/lzip-1.21-i586+1’. Assuming that the main binary for lzip is under ‘/home/selk/pkgs/lzip-1.21-i586+1/usr/bin/’ the target for "usr/bin" will be created at ‘/home/selk’. Considering that you have exported the PATH as ‘${HOME}/usr/bin’, now the system is able to see the recent lzip.

Installing from a list of packages using standard input:

cat FILELIST.txt | qi -i -

The list of packages must contain full path names to be passed in the installation, e.g.: /var/cache/qi/packages/x86_64/devel/tcl-8.6.9-x86_64+1.tlz /var/cache/qi/packages/x86_64/devel/tk-8.6.9.1-x86_64+1.tlz /var/cache/qi/packages/x86_64/devel/vala-0.42.3-x86_64+1.tlz

4.3 Removing packages

To remove a package, simply type:

qi -d xz-5.2.4-i586+1.tlz

Delete mode will match the package name using ‘${packagedir}’ as prefix. For example, if the value of ‘${packagedir}’ is set to /usr/local/pkgs, this will be equal to:

qi -d /usr/local/pkgs/xz-5.2.4-i586+1

Detailed output (very verbose):

qi -d -v /usr/local/pkgs/xz-5.2.4-i586+1

A second -v gives more.

By default the delete mode does not preserve a package directory after removing its links from ‘${targetdir}’, but this behavior can be changed if the -k option is passed:

qi -d -k /usr/local/pkgs/lzip-1.21-i586+1

This means that the links to the package can be reactivated, later:

cd /usr/local/pkgs && graft -i lzip-1.21-i586+1

Removing package from a different location:

qi -r /home/cthulhu -P /pkgs -t / -d xz-5.2.4-i586+1

Removing a package using standard input:

echo "vala-0.42.3-x86_64+1" | qi -d -

This will match with the package directory.

4.4 Upgrading packages

The upgrade mode inherits the properties of the installation and removal process. To make sure that a package is updated, the package is installed in a temporary directory taking ‘${packagedir}’ into account. Once the incoming package is pre-installed, qi can proceed to search and delete packages that have the same name (considered as previous ones). Finally, the package is re-installed at its final location and the temporary directory is removed.

To upgrade a package, just type:

qi -u gcc-9.0.1-i586+1.tlz

This will proceed to update "gcc-9.0.1-i586+1" removing other versions of "gcc" (if any).

If you want to keep the package directories of versions found during the upgrade process, just pass:

qi -u -k gcc-9.0.1-i586+1.tlz

To see the upgrade process (very verbose):

qi -u -v gcc-9.0.1-i586+1.tlz

A second -v gives more.

To force the upgrade of an existing package:

qi -u -f gcc-9.0.1-i586+1.tlz

4.4.1 Package blacklist

To implement general package facilities, either to install, remove or maintain the hierarchy of packages in a clean manner, qi makes use of the pruning operation via graft(1):

There is a risk if those are crucial packages for the proper functioning of the system, because it implies the deactivation of symbolic from the target directory, especially when transitioning an incoming package into its final location during upgrade.

A blacklist of package names has been devised for the case where a user decides to upgrade all packages in the system, or just the crucial ones, such as the C library.

The blacklist is related to the upgrade mode only, consists in installing a package instead of updating it or removing previous versions of it; the content of the package will be updated over the existing content at ‘${packagedir}’, while the existing links from ‘${targetdir}’ will be preserved. A pruning of links will be carried out in order to re-link possible differences with the recent content, this helps to avoid leaving dead links in the target directory.

Since the upgrade mode is also used to install a new package, the mechanism for blacklist is to install a declared package if it does not already exist. If it already exists, it is verified that the binary package is newer than the package directory in order to perform an update.

Package names for the blacklist can be set from the configuration file.

5 Recipes

A recipe is a file telling qi what to do. Most often, the recipe tells qi how to build a binary package from a source tarball.

A recipe has two parts: a list of variable definitions and a list of sections. By convention, the syntax of a section is:

section_name()
{
    section lines
}

The section name is followed by parentheses, one newline and an opening brace. The line finishing the section contains just a closing brace. The section names or the function names currently recognized are ‘build’.

The ‘build’ section is an augmented shell script. This is the main section (or shell function) which contains the instructions to build and produce a package.

5.1 Variables

A "variable" is a shell variable defined either in qirc or in a recipe to represent a string of text, called the variable’s "value". These values are substituted by explicit request in the definitions of other variables or in calls to external commands.

Variables can represent lists of file names, options to pass to compilers, programs to run, directories to look in for source files, directories to write output to, or anything else you can imagine.

Definitions of variables in qi have four levels of precedence. Options which define variables from the command-line override those specified in the qirc file, while variables defined in the recipe override those specified in qirc, taking priority over those variables set by command-line options. Finally, the variables have default values if they are not defined anywhere.

Options that set variables through the command-line can only reference variables defined in qirc and variables with default values.

Definitions of variables in qirc can only reference variables previously defined in qirc and variables with default values.

Definitions of variables in the recipe can only reference variables set by the command-line, variables previously defined in the recipe, variables defined in qirc, and variables with default values.

5.2 Special variables

There are variables which can only be set using the command line options or via qirc, there are other special variables which can be defined or redefined in a recipe. See the following definitions:

‘outdir’ is the directory where the packages produced are written. This variable can not be redefined in the recipe. Default sets to ‘/var/cache/qi/packages’.

‘worktree’ is the working tree where archives, patches, and recipes are expected. This variable can not be redefined in the recipe. Default sets to ‘/usr/src/qi’.

‘tardir’ is defined in the recipe to the directory where the tarball containing the source can be found. The full name of the tarball is composed as ‘${tardir}/$tarname’. Its value is available in the recipe as ‘${tardir}’; a value of . for ‘tardir’ sets it to the value of CWD (Current Working Directory), this is where the recipe lives.

‘arch’ is the architecture to compose the package name. Its value is available in the recipe as ‘${arch}’. Default value is the output of ‘uname -m’.

‘jobs’ is the number of parallel jobs to pass to the compiler. Its value is available in the recipe as ‘${jobs}’. The default value is 1.

The two variables ‘${srcdir}’ and ‘${destdir}’ can be set in the recipe, as any other variable, but if they are not, qi uses default values for them when building a package.

‘srcdir’ contains the source code to be compiled, and defaults to ‘${program}-${version}’. ‘destdir’ is the place where the built package will be installed, and defaults to ‘${TMPDIR}/package-${program}’.

If ‘pkgname’ is left undefined, the special variable ‘program’ is assigned by default. If ‘pkgversion’ is left undefined, the special variable ‘version’ is assigned by default.

‘pkgname’ and ‘pkgversion’ along with: ‘version’, ‘arch’, and ‘release’ are used to produce the name of the package in the form: ‘${pkgname}-${pkgversion}-${arch}+${release}.tlz’

A special variable called ‘replace’ can be used to declare package names that will be replaced at the time of installation.

A typical recipe contains the following variables:

• ‘program’: software name.

  It matches the source name. It is also used to compose the name of the package if ‘${pkgname}’ is not specified.

• ‘version’: software version.

  It matches the source name. It is also used to compose the version of the package if ‘${pkgversion}’ is not specified.

• ‘arch’: software architecture.

  It is used to compose the architecture of the package in which it is build.

• ‘release’: release number.

  This is used to reflect the release number of the package. It is recommended to increase this number after any significant change in the recipe or post-install script.

Obtaining sources over the network must be declared in the recipe using the ‘fetch’ variable. Use double quotes for separated values.

The variables ‘netget’ and ‘rsync’ can be defined in qirc to establish a network downloader in order to get the sources. If they are not defined, qi uses default values:

‘netget’ is the general network downloader tool, defaults sets to ‘wget -c -w1 -t3 --no-check-certificate’.

‘rsync’ is the network tool for sources containing the prefix for the RSYNC protocol, default sets to ‘rsync -v -a -L -z -i --progress’.

The variable ‘description’ is used to print the package description when a package is installed.

A description has two parts: a brief description, and a long description. By convention, the syntax of ‘description’ is:

description="
Brief description.

Long description.
"

The first line of the value represented is a brief description of the software (called "blurb"). A blank line separates the brief description from the long description, which should contain a more descriptive description of the software.

An example looks like:

description="
The GNU core utilities.

The GNU core utilities are the basic file, shell and text manipulation
utilities of the GNU operating system.  These are the core utilities
which are expected to exist on every operating system.
"

Please consider a length limit of 78 characters as maximum, because the same one would be used on the meta file creation. See The meta file section.

The ‘homepage’ variable is used to declare the main site or home page:

homepage=http://www.gnu.org/software/gcc

The variable ‘license’ is used for license information². Some code in the program can be covered by license A, license B, or license C. For "separate licensing" or "heterogeneous licensing", we suggest using | for a disjunction, & for a conjunction (if that ever happens in a significant way), and comma for heterogeneous licensing. Comma would have lower precedence, plus added special terms.

license="LGPL, GPL | Artistic + added permission"

5.3 Writing recipes

Originally, qi was designed for the version 3 of Dragora GNU/Linux (this does not mean that you can’t use it in another distribution, just that if you do you will need to test it for your selves). To aid this here are some references to well written recipes:

http://git.savannah.nongnu.org/cgit/dragora.git/tree/recipes. http://notabug.org/dragora/dragora/src/master/recipes.

You can also check the "doc" directory in the distribution sources of qi for some examples.

5.4 Building packages

A recipe is any valid regular file. Qi sets priorities for reading a recipe, the order in which qi looks for a recipe is:

1. Current working directory.
2. If the specified path name does not contain "recipe" as the last component. Qi will complete it by adding "recipe" to the path name.
3. If the recipe is not in the current working directory, it will be searched under ‘${worktree}/recipes’. The last component will be completed adding "recipe" to the specified path name.

To build a single package, type:

qi -b x-apps/xterm

Multiple jobs can be passed to the compiler to speed up the build process:

qi -b -j3 x-apps/xterm

Update or install the package produced (if it is not already installed) when finish:

qi -b -j3 -u x-apps/xterm

Only process a recipe but do not create the binary package:

qi -b -n dict/aspell

The options -i or -u have no effect when -n is given.

This can be useful to inspect the build process of recipe:

qi -b -k -n dict/aspell 2>&1 | tee aspell-buildlog.txt

The -k option could preserve the source directory and the destination directory for later inspection. A log file of the build process will be created redirecting both, standard error and standard output to tee(1).

5.5 Variables from the environment

Qi has environment variables which can be used at build time:

The variable TMPDIR sets the temporary directory for sources, which is used for package extractions (see Examining packages) and is prepended to the value of ‘${srcdir}’ and ‘${destdir}’ in build mode. By convention its default value is equal to ‘/usr/src/qi/build’.

The variables QICFLAGS, QICXXFLAGS, and QILDFLAGS have no effect by default. The environment variables such as CFLAGS, CXXFLAGS, and LDFLAGS are unset at compile time:

Recommended practice is to set variables in the command line of ‘configure’ or make(1) instead of exporting to the environment. As follows:

Variables not defined in a site shell script can be set in the environment passed to configure. However, some packages may run configure again during the build, and the customized values of these variables may be lost. In order to avoid this problem, you should set them in the configure command line, using ‘VAR=value’. For example:

./configure CC=/usr/local2/bin/gcc

http://gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/html_node/Defining-Variables.html

Indeed, while configure can notice the definition of CC in ‘./configure CC=bizarre-cc’, it is impossible to notice it in ‘CC=bizarre-cc ./configure’, which, unfortunately, is what most users do.

[...]

configure: error: changes in the environment can compromise the build.

http://gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/html_node/Setting-Output-Variables.html

It is not wise for makefiles to depend for their functioning on environment variables set up outside their control, since this would cause different users to get different results from the same makefile. This is against the whole purpose of most makefiles.

http://gnu.org/software/make/manual/make.html#Environment

5.6 The meta file

The "meta file" is a regular file created during the build mode, it contains information about the package such as package name, package version, architecture, release, fetch address, description, and other minor data extracted from processed recipes. The name of the file is generated as ‘${full_pkgname}.tlz.txt’, and its purpose is to reflect essential information to the user without having to look inside the package content. The file format is also intended to be imported from other scripts.

The content of a meta file looks like:

#
# The Bourne Again SHell.
#
# Bash is an sh-compatible shell that incorporates useful features from
# the Korn shell (ksh) and C shell (csh).  It is intended to conform to
# the IEEE POSIX P1003.2/ISO 9945.2 shell and tools standard.
#
# It offers functional improvements over sh for both programming and
# interactive use.
#

QICFLAGS="-g0 -Os -mtune=generic -pipe"
QICXXFLAGS="-g0 -Os -mtune=generic -pipe"
QILDFLAGS="-s"
pkgname=bash
pkgversion=5.0
arch=x86_64
release=1
blurb="The Bourne Again SHell."
homepage="http://www.gnu.org/software/bash"
license="GPLv3+"
fetch="ftp://ftp.gnu.org/gnu/bash/bash-5.0.tar.gz"
replace=""

Package descriptions are extracted from the variable ‘description’ where each line is interpreted literally and pre-formatted to fit in (exactly) 80 columns, plus the character ‘#’ and a space is prefixed to every line.

In addition to the Special variables, there are implicit variables such as ‘blurb’:

The ‘blurb’ variable is related to the special variable ‘description’. Its value is composed using the first (substantial) line of ‘description’, mentioned as the "brief description".

6 Order files

The order mode has the purpose of resolving the build order through .order files. An order file contains a list of recipe names, by default does not perform any action other than to print a resolved list in descending order. For example, if a depends on b and c, and c depends on b as well, the file might look like:

a: c b
b:
c: b

Each letter represents a recipe name, complete dependencies for the first recipe name are listed in descending order, which is printed from right to left, and removed from left to right:

OUTPUT

b
c
a

Blank lines, colons and parentheses are simply ignored. Comment lines beginning with ‘#’ are allowed.

An order file could be used to build a series of packages, for example, if the content is:

# Image handling libraries

libs/libjpeg-turbo: devel/nasm
x-libs/jasper: libs/libjpeg-turbo
libs/tiff: libs/libjpeg-turbo

To proceed with each recipe, we can type:

qi -o imglibs.order | qi -b -i -

The output of ‘qi -o imglibs.order’ tells to qi in which order it should build the recipes:

devel/nasm
libs/libjpeg-turbo
x-libs/jasper
libs/tiff

7 Creating packages

The "creation mode" is an internal function of qi to make new Qi compatible compatible packages, the creation mode is selected by the -c option. A package is produced using the contents of the Current Directory, and the package file is written out.

Usage: qi -c [OUTPUT/packagename.tlz]...

The argument for the file name to be written must contain a fully qualified named directory as the output directory where the package produced will be written. The file name should be composed using the full name: name-version-architecture+release.tlz

EXAMPLE

cd /usr/local/pkgs
cd claws-mail-3.17.1-x86_64+1
qi -c /var/cache/qi/packages/x86_64/local/claws-mail-3.17.1-x86_64+1.tlz

In this case, the package "claws-mail-3.17.1-x86_64+1.tlz" will be written into ‘/var/cache/qi/packages/x86_64/local/’.

All packages produced are complemented by a checksum file (.sha256).

8 Examining packages

The "extraction mode" serves to examine binary packages for debugging purposes. The extraction mode is selected by the -x option. It decompresses a package into a single directory, verifying its integrity and preserving its properties.

Usage: qi -x [packagename.tlz]...

EXAMPLE

qi -x mksh-R56c-x86_64+1.tlz

This action will put the content of "mksh-R56c-x86_64+1.tlz" into a single directory, this will be a private directory for the user who

requested the action, creation mode will be equal to u=rwx,g=,o= (0700). The package content will reside on this location, default mask to deploy the content will be equal to u=rwx,g=rwx,o=rwx (0000).

The creation of the custom directory is influenced by the value of the TMPDIR variable.

9 Exit status

All the exit codes are described in this chapter.

‘0’

Successful completion (no errors).

‘1’

Minor common errors:

• - Help usage on illegal options or required arguments.
• - Program needed by qi (prerequisite) is not available.

‘2’

Command execution error:

This code is used to return the evaluation of external commands and shell arguments in case of error.

‘3’

Integrity check error for compressed files.

Compressed files means:

• - Tarball files from tar(1). Supported extensions: .tar, .tar.gz, .tgz, .tar.Z, .tar.bz2, .tbz2, .tbz, .tar.xz, .txz
• - Tarball files from tarlz(1). Supported extensions: .tar.lz, .tlz
• - Zip files from unzip(1). Supported extensions: .zip, .ZIP
• - Gzip files from gzip(1). Supported extensions: .gz, .Z
• - Bzip2 files from bzip2(1). Supported extensions: .bz2
• - Lzip files from lzip(1). Supported extensions: .lz
• - Xz files from xz(1). Supported extensions: .xz

‘4’

File empty, not regular, or expected.

Commonly, it is expected:

• - An argument for the mode of operation.
• - A readable file or directory.
• - A binary package (.tlz).
• - A valid recipe.
• - An order file (.order).
• - A protocol supported by the network downloader tool.
• - A checksum file (.sha256).

‘5’

Empty or not defined variable:

This code is used to report empty or undefined variables; usually, variables coming from a recipe or assigned arrays that are tested.

‘6’

Package already installed:

The package directory for an incoming .tlz package already exists.

‘10’

Network manager error:

This code is used if the network downloader tool fails for some reason.

Index