This is about [[!tails_ticket 5926]].

[[!toc levels=3]]

<a id="todo"></a>

# TODO

1. design documentation: [[!tails_ticket 11447]]

2. misc: see subtasks of [[!tails_ticket 5926]]

# Bonus for later

## Handle handle ever-growing `references.db`

This is about [[!debbug 823629]]:: if `references.db` doesn't fit in
the memory disk cache, then at least our GC process gets very slow);
the visible consequence would be: long periods of heavy disk read, and
much slower snapshots expiration process; so we added an Icinga2 check
on the file size itself. When this problem occurs, our options are:

   * add more RAM to the VM if that's still feasible and reasonable
     (likely not)
   * reset the whole `debian` repository to an empty state (simple
     and big hammer, painful as it requires lots of coordination
     with developers)
   * compact the DB with `db5.3_dump` + `db5.3_load`: it got our
     big `debian/db/references.db` down from 5.4 GB to 1.5 GB
   * compact the DB with `DB->compact()`, using
     [a script we wrote](https://git-tails.immerda.ch/puppet-tails/tree/files/reprepro/snapshots/time_based/tails-compact-reprepro-db)
     (<https://docs.oracle.com/cd/E17076_02/html/programmer_reference/am_misc_diskspace.html>,
     <https://groups.google.com/forum/#!topic/memcachedb/uiSvgStzYNY>,
     <https://docs.oracle.com/cd/E17076_02/html/api_reference/C/dbcompact.html>):
     works on small databases, but on our big `debian` the file
     doesn't shrink

## Miscellaneous

If the chosen mirroring/snapshoting tool supported re-using the Debian
signature (e.g. <https://github.com/smira/aptly/issues/37>) then we
would only have to sign ourselves the snapshots for which need to
modify `Release` — that is: when we bump (too long freeze) or remove
(at release time) `Valid-Until` — which happens rarely and can be done
manually ⇒ we can avoid storing the signing key on an online server.

We might want to use reprepro's `Tracking:` feature in
`conf/distributions` once it's stabilized.

There's a race condition when updating a local mirror with `reprepro
update`: if it's not finished before the next dinstall + mirror sync'
end, then files `reprepro` wants to download can have disappeared from
the remote mirror, and `reprepro update` will fail (exit code = 255).
So, when the first run exits with exit code 255, let's ignore the
error and run `reprepro update` a second time.

## Handle full disk

"Reprepro uses berkeley db, which was a big mistake. The most
annoying problem not yet worked around is database corruption when
the disk runs out of space. (Luckily if it happens while
downloading packages while updating, only the files database is
affected, which is easy (though time consuming) to rebuild, see
recovery file in the documentation). *Ideally put the database on
another partition to avoid that.*" (emphasis mine, from
[reprepro(1)](https://mirrorer.alioth.debian.org/reprepro.1.html#BUGS))

Note: we have an Icinga2 check monitoring the filesystem that hosts
our APT snapshots, so in theory we should not experience this situation.

## Custom APT repository

Our custom APT repository (<http://deb.tails.boum.org/>) is not part of
our APT snapshots system: it's not needed, since we already
have a process to manage it, including creating snapshots labeled with
the Git tag.

However, longer-term, ideally we would integrate it into the new
system. It will require quite some infrastructure and code, if we want
to avoid making the release process more painful (e.g. it would be nice
if this didn't require waiting up to 6 hours until the next time-based
snapshot of our custom APT repository is created, between the time we
upload a package to it, and when we can build an ISO with it; we could
solve this by automatically creating a new snapshot whenever an APT
suite corresponding to a release branch is updated).

# Discarded

## "Remote" APT repository snapshots

... i.e. using snapshot.debian.org directly, instead of mirroring the
files ourselves.

Discarded because:

* not substantially simpler than our other designs;
* having to re-implement `Valid-Until` checking is scary;
* too much reliance on an external service.

frozen mode = when building from a tag => use snapshot.d.o with
a timestamp manually set in Git => need code that tells us what's the
dinstall timestamp used at some point during a validated build (racy
but no big deal; can kill the race condition by using a local mirror
whose update is disabled during builds)

regular mode = otherwise => use ftp.us.d.o

* Directly use snapshot.d.o + dinstall ID
  - basically replaces e.g. aptly's snapshot / "reprepro pull in new
    suite" feature
  - The fastest possible way to do a new snapshot, since we don't have
    to store nor pull anything at all.
  - Doesn't introduce a database we have to maintain and trust
    software not to ever corrupt it.
  - the dinstall ID that a given mirror was last updated can be
    retrieved from that mirror, e.g. `Archive serial` in
    <http://ftp.fr.debian.org/debian/project/trace/ftp-master.debian.org>
  - Blocker: `Valid-Until` can be invalid:
    * If we don't bump the dinstall ID at least once a week as part of
      the normal development process. Seems impractical (e.g.
      we sometimes freeze for more than a week) and too rigid.
    * When rebuilding from an old tag (old > a week).
  - But do we want to depend on snapshot.d.o that much?
    * Served from two different locations.
    * Ask weasel if we can go this way. Make it clear how much we care
      about _old_ data, e.g.:
      - For reproducible rebuild check we only care about re-building
        the last release, or the last few releases.
      - GPL requires distributing the source for at least 3 years
        after we stop distributing the binaries.
  - Whonix uses that, go look/ask for pros/cons they've seen.
  - other repos e.g. deb.tpo; we can probably handle it in a very
    ad-hoc and lightweight way, by importing the packages we want into
    our own Tails-specific APT suites, or with reprepro's mirroring
    (`pull`) feature.

- to avoid relying on browsing <http://snapshot.debian.org/> for
  getting the dinstall timestamp we'll stick into Git, we need
  a script that does the browsing _and_ validates that the determined
  timestamp is not too far away in the past.

## Partial APT repository snapshots

Discarded because:

* it raises a tricky chicken'n'egg problem: managing the list of
  packages that will be needed to build a given Git branch, and
  maintaining the set of partial APT repository snapshots that these
  branches need;
* partially snapshot'ing live APT repositories (e.g.
  security.debian.org) is racy: between the time when we build an ISO
  to get the list of packages we need to import, and the time we
  actually import them, files can have disappeared on the mirrors.

 - pro: faster sync ⇒ faster snapshots ⇒ shorter time to remediation
   However, we can have something similar with full snapshots, if we
   continuously update a temporary snapshot, and then when we need it
   we only have to stick some label onto it.
 - cons: more complex... except perhaps if we want to optimize time to
   remediation for full snapshots as described above.

### Snapshots name

For partial mirroring, their name must contain:

* Debian origin (`debian`, `debian-security`)
* Debian distribution (`sid`, `jessie`, `jessie-backports`, etc.)
* name of the Tails Git release/base branch that needs this set of
  packages

## Replace index expiration check

For the remote snapshots (snapshot.d.o) solution, we _have_ to do
that. For partial and full archive snapshots, this is optional: the
only advantage is that it allows us to _not_ periodically update
`Valid-Until` and signature.

One "solution" would be to replace `Acquire::Check-Valid-Until`:

 - runtime: we point APT sources to the regular Debian archive, no
   need to disable `Acquire::Check-Valid-Until`, we're good.
 - ISO build time: we know when we've frozen ⇒ we can tell APT not to
   do that check, and check the Release files ourselves based on the
   additional info and constraints we have; a bit risky, no right to
   fail, but not totally scary; so draft a security discussion, then
   have it reviewed

## dak, britney, merge-o-matic, debile, etc.

Overkill, and not really meant to address our needs.
Let's instead write our own :P

## Handling freeze exceptions by backporting

Another option, instead of adding/removing temporary APT pinning,
would be to backport the package we want to upgrade, and make it so it
has a version greater than the one in the time-based snapshot used by
the frozen release branch, and lower than the one in more recent
time-based snapshots. This means building and uploading the package to
the relevant custom APT suite. This is appealing, because it doesn't
require any cleanup: the upgraded package will automatically be
superseded as soon as it can be. However:

 * we would not benefit from Debian features like reproducible builds;
 * it requires either manual work and bandwidth every time, or setting
   up and maintaining infrastructure to automate the whole thing;
 * the fact that the changes *have to* go through Git, with the APT
   pinning option, helps enforcing our review'n'merge processes; one
   can do it by the book with the custom backport option too, by going
   through a topic branch and `config/APT_overlays.d/`, but it still
   conveys less historical information through Git than the APT
   pinning option;
 * anonym NACK'ed this ("Please do not underestimate the added
   overhead of having to rebuild packages for trivialities like
   this"), and intrigeri agreed.
