[[!meta title="Automated builds specification"]]


This blueprint helps to keep track of the discussion on the mailing
list, and is attached to [[!tails_ticket 8655]]
to specify how to implement [[!tails_ticket 6196]] ("Build all active
branches").

Some metrics about the number of branches merged per releases are
available on the [[dedicated statistics page|autobuild_stats]].

Our Jenkins now has the Global Build Stats plugin live, Tails core developers
[have access to the metrics](https://jenkins.tails.boum.org/plugin/global-build-stats/).

[[!toc levels=2]]

# Question to discuss

## Which branches we want to build?

We already build the base branches (_stable_, _testing_, _devel_ and
_experimental_) + _feature/jessie_.

The questions raised is mostly concern the _feature/*_ and _bugfix/*_ branches
(so _topic branches_), as well as the _doc/*_ and _test/*_ branches.

Given an ISO build takes around 30-45 minutes on lizard (worst case),
given two builders lizard will be able to build something like 64-96
ISOs a day.

Developers can easily add a branch back to the automated builds
whenever it has been removed from the list (for example after
a release) by merging its base branch into it.

They also can at the moment manually trigger a build if they uploaded to
a APT suite:

1. if anyone really want to trigger an immediate rebuild, they can do
   it manually in the Jenkins interface (people who have upload
   rights to our APT repo also have powerful-enough access to Jenkins
   to trigger builds);
2. the proposal says that all active branches are built daily, in
   addition to post-Git-push => worst case, the branch will be
   rebuilt within 24h;
3. if in a hurry, or for whatever reason, you can still force
   a rebuild by pushing a dummy commit (ugly, but it works).

Proposal1:

* branches which are not merged into master, devel, stable and testing
* but had new commits or Debian package uploaded since the previous release

<a id="how-to-build-it"></a>

## How to build it

A topic branch may be lagging behind the base branch it's based upon.
What we're interested in is generally *not* whether a (possibly
outdated) topic branch builds fine, but whether it would build fine
**once merged into its base branch**:

* that's critical from a reviewer's perspective: what they have to
  evaluate is the result of the merge, not the state of a topic branch
  forked N weeks ago from its base branch, that possibly has diverged
  since then;
* that's important from a developer's perspective: this is how they
  will notice if incompatible changes have landed into the base branch
  since last time they worked on their topic branch.

Hence, when building topic branch F, we need to build from branch F
*once merged into* branch B. However, this merge must only be done
*locally*, at least because Jenkins doesn't have push access to our
Git repo.

Here, *locally* means: in Jenkins own temporary Git checkout.

The exact direction of the merge (B->F vs. F->B) should not matter
given how Git merge works, if we got it clearly. We'll see.

This locally-merge-before-building process requires [[!tails_ticket
8654]] to be implemented, otherwise we can't easily merge branches
*locally* without affecting the state of our production APT repo.

## When to build it

Define the regularity we want to build topic branches, apart from being built
on Git push or new Debian package upload.

Note that we will have to plug that in automatic tests when they will be
deployed.

Proposal 1: A build a day.


## Notifications

When to notify who? And how to notify them?

Proposal 1:

Email will be the main interface.

* For base branches, notify the RM.
* For topic branches, notify the author of the last commit.

Note that this proposal doesn't take into consideration how to notify
when the branch is built because of a Debian package upload.

An alternative for topic branches we might want to use in the future is to
notify all authors of the topic branch since it deviated from the base branch:

    git log --pretty="format:%an <%ae>" ${BASE_BRANCH}.. | sort -u

# Scenarios

In the following scenario:

0. topic branches are named branch F
0. base branches are named branch B
0. builds are ran on merges which don't raise a conflict. If the merge raises a
conflict, then the topic branch's developer should take care of resolving it.


## Scenario 1 : reviewer

    As a reviewer
    When I'm asked to review branch F into branch B
    Then I need to know if branch F builds fine
      once locally merged into branch B (fresh results!)
    So, if there is no such fresh build available
       Then I manually trigger a build of branch F
    And if the build succeeded
      The resulting ISO must be made available to me
      The pkg list must be made available to me
      The Redmine ticket should be notified
    Otherwise if it fails the developer who proposed the merge must be notified
      And the developer *needs* to see the build logs
      And the ticket should be reassigned to the branch submitter
      And QA check should be set to "Dev needed"

Bonus:

* Notify the manual build requester of the build results. Depends on
  using Jenkins internal authentication system, and may be complicated
  if it doesn't attach email address info to each user (apparently the
  Email-ext plugin just builds the email address by concatenating
  login, `@` and a fixed domain name -- this could be worked around
  with email aliases hosted somewhere on our infrastructure).
* Make it easy for the reviewer to know whether the last build of
  branch F is current. Or, better (?), trigger rebuilds of all topic
  branches upon modifications (possibly == rebuild) on their
  base branch.

## Scenario 2 : developer

    As a developer who has the commit bit
    When I'm working on branch F
    Then I need to know if my branch builds after I've pushed and it
        is has been locally merged in base branch B
    And I need to know if my branch build is broken by something else
       possibly weeks after my last commit (by e.g Debian changes,
       changes in branch B, ...)
    And if the build succeeded
      The resulting ISO must be made available to me
      The pkg list must be made available to me
      The Redmine ticket should be notified
    Otherwise if it fails I *need* to see the build logs
      And the developer who proposed the merge must be notified
      And the ticket should be reassigned to the branch submitter
      And QA check should be set to "Dev needed"


## Scenario 3 : RM

    As the current RM
    When working the full dev release cycle
    Then I need to know when a base branch fails to build
    And when this happens the build logs must be made available to me.


# Future ideas

This list other scenarios not part of the first deployement iteration, but we
might want to consider it in the future.

## Scenario 10

    As a Tails developer working on branch B
    When I upload a package to APT suite B
    Then I want to know if it broke the build ASAP

(same responsiveness as when pushing to git)
(acceptable workaround: being able to manually trigger a build.)


## Scenario 11

    As the current RM
    When I push new tag T on branch B
    Then I want the APT suite for tag T to be created
    And I want the APT suite B to be copied into the APT suite T
    And once this is done, I want a build from the checkout of tag T to be
      triggered
    And I want the squashfs sort file to be generated, and the diff sent to me


## Scenario 12

    As a Tails developer
    When the test suite is ran on the ISO build from my last commit
    I want to watch TV and see the test video in HTML5 from Tor Browser


## Scenario 13

    As a Tails developer
    When an ISO is build from my last commit
    I want to access it through remote desktop (VNC/Spice/...) over Tor

## Scenario 14

    As a Tails developer
    When I push a new commit or a new Debian package to a base branch
    I want the affected feature branches to be rebuilt with that change


# Statistics

As of 2015-02-02, there are 26 branches that would be automatically
built as part of the next 1.3 release, following the for now defined
criterias (above in this blueprint):

* feature/7779-revisit-touchpad-settings
* feature/6992-whisperback-email-address
* bugfix/8714-tor-is-ready-robustness
* bugfix/8680-git-without-polipo
* feature/8719-mount-output-in-bug-reports
* feature/6241-gnupg2
* feature/8725-remove-vagrant-bootstrap-cache
* bugfix/8715-build-system-independent-APT-sources
* feature/7756-reintroduce-whisperback
* bugfix/8699-only-create-timestamps-in-Jenkins
* feature/8740-new-signing-key-phase-2
* feature/8665-remove-adblock
* bugfix/8756-repair-local-packages
* feature/7530-docker_anonym
* feature/7530-docker-with-apt-cacher-ng
* feature/7963-background-color
* feature/8491-live-additional-software-in-whisperback
* feature/7530-docker
* feature/linux-3.18
* feature/torbrowser-alpha
* bugfix/8747-update-tails-apt-repo-signing-key
* feature/8726-use-homogenous-Debian-mirrors-at-build-time
* feature/5525-sandbox-web-browser
* feature/7752-keyringer
* feature/6739-install-electrum
* bugfix/quote-wrappers-arguments
