dragora/jul/design.txt

Lucas Skoeldqvist <frusen@dragora.org>
2016-11-30 rev 1: initial write up 

1 INTRODUCTION
==============

Dragora does have a system for managing packages already (pkgsystem[1] in
Dragora version 2 and qi[2] in version 3). Both of them can build, install,
remove, and upgrade packages, but with limited flexibility; they only work
with local packages on the system. There is no way to search for available
packages, check for upgrades, or fetch packages.

This is where jul comes in. Jul uses the existing package managers and adds
aditional functionality on top. There is a working yet incomplete version
available for use today on Dragora 2.

One can think of pkgsystem/qi as dpkg and jul as apt-get (in Debian) but with
a simpler design.

[1] http://www.dragora.org/repo.fsl/doc/trunk/www/dragora2/guides/pkgmanager.md
[2] http://www.dragora.org/repo.fsl/dir?name=qi

2 DESIGN
========

Jul is written in Tcl[1] and has a command line interface. The interface is
modeled after Fossil[2] but commands are named similar to that of pkgsystem
and qi. Some examples:

	$ jul search foo

	$ jul help add

	$ jul add -h

[1] https://tcl.tk/
[2] http://fossil-scm.org/

2.1 COMMANDS
============

The following commands are needed:

	1) add/install

	2) list

	3) search

	4) sync/update

	5) upgrade

	6) help

2.1.1 ADD/INSTALL
=================

These commands do the same thing; adds a package to system. Jul will do a
search for the given package(s) and list the matches. One can then choose from
the list of packages which package(s) to install.

In jul 0.5 you can only add one package at a time, and a list of matches makes
sense. But it is not clear how this would work when multiple packages are
specified on the command line.

2.1.2 LIST
==========

This command lists installed packages. It can have options to only list
packages from certain repositories (see section 3) or match against a search
term.

2.1.3 SEARCH
============

`search' takes on or more search-terms and looks for packages. By default it
searches in all enabled repositories but it could take an option to only search
in one or more repositories.

Currently searches in a local copy of the database of available packages. One
might want to do an update simultaneously (see section 4).

2.1.4 SYNC/UPDATE
=================

These commands fetch the latest changes from the repositories.

2.1.5 UPGRADE
=============

`upgrade' compares the installed packages with the ones in the database, and
if they are newer in the database, they can be upgraded.

Some useful options to this commands could be:

	1) --list/-l       only list available upgrade, don't prompt to upgrade

	2) --onlysec/-o    only list/upgrade packages that are vulnerable

2.1.6 HELP
==========

The `help' command is used to display help screens. It shows the same help
that the `-h' option does, so these command lines are equivalent:

	$ jul help

	$ jul -h

	$ jul help add

	$ jul add -h

3 REPOSITORIES
==============

A repository is a SQLite[1] database with a few tables. Currently (see section
3.2) it has support for basic package information plus descriptions.

Packages are added to the database and then the database is downloaded by jul
when doing a sync/update. The copy is stored in $HOME/.jul and used by most
commands.

Up until now I've manually (or via scripts) added package information to the
database. It is a lot of work and we need a better and simpler way...

[1] https://sqlite.org/

3.1 GENERATING DATABASES
========================

Qi introduces `.meta' files, which contain most of the information needed for
an entry in the database. A meta file could look something like this:

	name="foo"
	version="1.3"
	desc="Simple program"
	license="GPLv3+"
	order="bar zab foo"

If each package has one of these files a database can easily be generated by
parsing these files.

3.2 SQLITE SCHEMA
=================

CREATE TABLE repository (
    name text,
    url text,
    PRIMARY KEY(name)
);
CREATE TABLE description (
    name text,
    lang text,
    desc text,
    PRIMARY KEY(name)
);
CREATE TABLE location (
    name text,
    url text,
    PRIMARY KEY(name)
);
CREATE TABLE package (
    repo text,
    name text,
    version text,
    arch text,
    build integer,
    FOREIGN KEY(repo) REFERENCES repository,
    FOREIGN KEY(name) REFERENCES description
);

3.3 HOSTING AND SYNCHRONISING
=============================

One hosts a database simply by serving it via HTTP. The database on
gungre.ch is available at: https://gungre.ch/jul/gungre.db

(the databases contain one or more repositories)

Curl[1] is used for fetching so other protocols are supported as well.

Jul supports multiple databases so you could point at dragora.org for the
default repository, gungre.ch for community repositories, and so on.

It would be better if each host could synchronise the other hosts databases and
packages so if one goes down there are other mirrors available.

So each host has a directory tree with packages. The host then runs a script
(see section 3.4) each time packages are added (or using a cron job, or
something) to generate the database.

The hosts will then, maybe twice a day, or every hour, use rsync[2] to fetch
the new version of the database and the new packages.

[1] https://curl.haxx.se/
[2] https://rsync.samba.org/

3.4 VERSION COMPARISON
======================

The script that generates the databases should only add the latest packages
to the database. To achive this the different versions of a package need to be
compared.

To compare versions we need a specification how to version packages. For
example the author of a package might use the version 1.2-alpha1 and another
author might use the version 2.0a. We need to decide which style to use, to be
able to compare them.

All package managers that does version comparison does this, Gentoo's portage
is not an exception[1].

Besides the specification, some algorithms are needed. I'm currently
translating the algorithms from portage[2].

When the code is written it will be used for jul's `upgrade' command as well.

[1] https://dev.gentoo.org/~ulm/pms/head/pms.html#x1-300003.2
[2] https://dev.gentoo.org/~ulm/pms/head/pms.html#x1-310003.3

4 CONFIGURATION FILES
=====================

The configuration file is just a Tcl script where you can set some variables.

There is a system wide configuration file in `/etc' and a local configuration
file in the users home directory. Like most programs.

Examples of settings:

	1) which packages to show (archs)

	2) which repositories to use

	3) always run the sync/update command before most of the other
	   commands

Most, if not all, of the settings should be modified via the command line as
options.

5 DISCUSSION
============

A limited version of jul is available[1,2] today, and a few people in the
community including me use it a lot even if the most exiting features aren't
complete or even missing.

The most pressing thing at the moment is the code for version comparison. I'm
working on it but we need a version specification for it to work. I'm using
portage's version spec. temporarily.

This document was written in somewhat of a hurry and I'm sure things were left
out.

Please discuss on the mailing list and IRC so we have something ready for the
long awaited Dragora 3. :-)

[1] https://gungre.ch/dragora/repo/frusen/stable/jul/
[2] http://dragora.org/repo.fsl/dir?name=jul