[[!meta title="Automated tests specification"]]

*Ticket*: [[!tails_ticket 8667]]

This blueprint helps to keep track of the discussions and decisions
about the specification of automated tests ran in Tails' Jenkins on
the [[automatically build ISOs|autobuild_specs]] ([[!tails_ticket 5288]]).

[[!toc levels=3]]

# Facts

Running the full test suite on 1 isotester hosted on Lizard takes
around 8 hours.
We intend to run __4 isotesters__, so at the moment we would be able
to run __12 full test suites__ per day. ([[!tails_ticket 6094]] will
save ~25% on the full test suite runtime, but we'll keep on adding
more tests, so let's stick to the 8 hours estimate for now.)

We have 2 isobuilders on Lizard, that build a total of a bit less than
__400 ISOs/month__ (that's __an average of 13 ISOs/day__).
At the moment, we build __from 6 ISOs to 30-40 ISOs a day__, depending
on the activity.

We usually build the _stable_, _devel_, _experimental_,
_feature/jessie_ (+ _testing_ sometimes) and a bunch of other
branches.

This numbers are expected to grow when the automated builds will be
put in production. It's difficult to guess what would the maximum
number of builds per day be, but the minimum should be expected to
raise to in between __10/20 ISOs/day__.

# Things to keep in mind

There will probably be new hardware at the end of the year to deploy
more isotesters. If a machine is dedicated to that usage, we can throw
in faster CPUs and run the test suite on bare metal, which would
speed up the test process. That's [[!tails_ticket 9264]].

So in this discussion, we have to think to a deployment that might
have two iterations with different computational powers (and thus
different amounts of tests/day possible), and the defined
implementation should be modular enough to handle both of them without
too many changes.

# Questions

<a id="when-to-test-the-builds"></a>

## When to test the builds

It doesn't sound reasonable to run the full test suite on every ISO
automatically built by Jenkins, at least in the first iteration.

We __need to lessen the number of tests per day__. Probably even in
the second iteration, with the expected growth of the number of
automated builds.

At first, we'll try to run the full test suite on all of them. If our
infra can't cope with that, we have ideas written below to help in that.
The one that everyone seem to agree on is to run the full test suite for
base branches and features branches that are in ReadyForQA status in
Redmine.

We might want to play with job priorities, the same way we'll do with
the automated builds (see [[!tails_ticket 9760]]). Luckily, configuring
that for the builds will probably also work by itself for their automated
tests, as this jobs will be chained together.

### Current proposal

    For branch stable:
      Must test ISOs built daily and on every Git push;
        so that we're always ready to put out an emergency release;
    For the next scheduled release branch (either stable, testing, or devel):
      Must test ISOs built daily and on every Git push;
        so that we always know the state of the next release;
    For other branches:
      For branches with a `Ready for QA` ticket:
        Must test ISO built daily and on every Git push;
      Must test ISOs built on every Git push,
        with some rate-limiting if necessary;

<a id="how-to-run-the-tests"></a>

## How to run the tests

This section heavily depends on the discussion about the previous one.

To test a specific ISO, the automated test suite MUST have access to:

 * the corresponding build artifacts;
 * the commit ID of that build;
 * if applicable, the commit at which the base branch was at, when it
   was merged into the branch being built;
 * the ISO of the previous Tails release.

The automated test suite MUST be run in a clean environment, using a
fresh _--tmpdir_ for each run.

The automated test suite MUST be run on a freshly built ISO,
corresponding to the commit it tests.

When [[!tails_ticket 9515]] and friends will be resolved and a first
deployment of the automated test suite will clarify its need, we'll see
if it should be able to accept a treshold of failures for some features
before sending notifications. This could help if a scenario fails
e.g. because of a network congestion.

## Notifications

Email will be the main interface. Use the same settings than for
automated builds:

* For base branches, notify the RM.
* For topic branches, notify the author of the last commit.

When notifying Redmine, the use of the ReadyForQA custom field is not so
semantically correct, and may overlap with the dev usage of it. We will
probably have to add a new one for Jenkins to decide to test the branch
if we choose to test only the feature branch that are ready to be
reviewed.

## What kind of result shall be kept

The test suite produces different kind of artifacts: logfiles, screen
captures for failing steps, snapshots of the test VM, and also videos of
the running test session.

We can keep the video captures in the build artifacts, now that
[[!tails_ticket 10001]] is resolved.

Decision:

 * For green test suite run: keep the test logs (Jenkins natively do
   that).
 * For red test suite run: keep the screenshots and video captures, the
   logs and the pcap files.

In [[!tails_ticket 10155]] we calculated that we can probably keep the
video captures for a full release cycle. This will be refine is reality
claims the contrary after an evaluation.

# Scenarios

In the following scenario:

0. topic branches are named branch F
0. base branches are named branch B

We also assume that the following scenarios expectations only cover
the test suite artifacts, the ISO one being covered in the automated
builds specification.

## Scenario 1 : reviewer

    As a reviewer
    When I'm asked to review branch F into branch B
    Then I need to know if branch F passes the automated tests
      once locally merged into branch B and built (fresh results!)
    And if all the automated tests scenarios suceeded
      The resulting test logs must be made available to me
      The Redmine ticket should be notified
    Otherwise the developer who proposed the merge must be notified
      And the developer *needs* to see the test logs and screenshots
      And the ticket should be reassigned to the branch submitter
      And QA check should be set to "Dev needed"

## Scenario 2 : developer

    As a developer who has the commit bit
    When I'm working on branch F
    Then I need to know if my branch passes the automated tests suite
      once locally merged in base branch B
    And I need to know if my branch fails to pass the automated test
      suite because of an external change possibly weeks after my
      last commit (by e.g Debian changes, changes in branch B, ...)
    And if my branch passes the automated test suite
      The resulting test logs must be made available to me
      The Redmine ticket should be notified
    Otherwise I *need* to see the build logs and screenshots
      And I must be notified
      And the ticket should be reassigned to me if needed
      And QA check should be set to "Dev needed"


## Scenario 3 : RM

    As the current RM
    When working the full dev release cycle
    Then I need to know if a base branch does not pass the automated
      test suite
    And when this happens the build logs and screenshots must be made
      available to me.


# Future ideas

## Cutting the number of run per day

For feature branches, we could run the full test suite only on the daily
builds or on ReadyForQA ones, and either only the automated tests
related to the branch on every git push, and/or a subset of the whole
test suite.

We can also maybe find more ways to split the automated test suite in
faster subsets of features depending on the context, define priorities
for built ISO and/or tests.

The automated test suite could be able to run features in parallel
for a single automated build ISO. This way, if more than one isotester
are idle, it can use several of them to test an ISO faster.

The automated suite then should be able when there are more than one ISO
queued for testing to fairly distribute the parallelizing of their
features.

The automated test suite also should not allocate all the isotesters for
one ISO when others are waiting to be tested.

## Scenarios

Folowing is a list of other scenarios that we have also envisaged for
the second iteration of the automated builds deployment, as they are
really tightened.

### Scenario 10

    As a Tails developer working on branch B
    When I upload a package to APT suite B
    Then I want to know if the automated test suite passes

(same responsiveness as when pushing to git)
(acceptable workaround: being able to manually trigger a test suite.)


### Scenario 11

    As the current RM
    When I push new tag T on branch B
    Then I want the APT suite for tag T to be created
    And I want the APT suite B to be copied into the APT suite T
    And once this is done, I want the automated test suite to be run
      on the ISO build from the checkout of tag T


### Scenario 12

    As a Tails developer
    When the test suite is ran on the ISO build from my last commit
    I want to watch TV and see the test video in HTML5 from the Tor Browser


### Scenario 14

    As a Tails developer
    When I push a new commit or a new Debian package to a base branch
    I want the affected feature branches ISO to be tested with that change
