[[!meta title="System administrators"]]

[[!toc levels=2]]

<a id="goals"></a>

# Goals

The Tails system administrators set up and maintain the infrastructure
that supports the development and operations of Tails. We aim at
making the life of Tails contributors easier, and to improve the quality of
the Tails releases.

<a id="principles"></a>

# Principles

## Infrastructure as code

We want to treat system administration like a (free) software
development project:

* We want to enable people to participate without needing an account
  on the Tails servers.
* We want to review the changes that are applied to our systems.
* We want to be able to easily reproduce our systems via
  automatic deployment.
* We want to share knowledge with other people.

This is why we try to publish as much as possible of our systems
configuration, and to manage our whole infrastructure with
configuration management tools. That is, without needing to log
into hosts.

## Free Software

We use Free Software, as defined by the [Debian Free Software
Guidelines](https://www.debian.org/social_contract#guidelines).  
The firmware our systems might need are the only exception to
this rule.

## Relationships with upstream

The [[principles used by the broader Tails
project|contribute/relationship_with_upstream]] also apply for
system administration.

<a id="duties"></a>

# Duties

## In general

As said above, "set up and maintain the infrastructure". This implies
for example:

* dealing with hardware purchase, upgrades and failures;
* upgrading our systems to a new version of Debian.

## During sysadmin shifts

* create Git repositories when requested
* update access control lists to resources we manage, as requested by
  the corresponding teams
* keep systems up-to-date, reboot them as needed
* keep backups up-to-date
* keep Puppet modules up-to-date wrt. upstream changes
* keep Jenkins plugins up-to-date, by upgrading any plugin that satisfies
  at least one of these conditions:
   - only brings security fixes
   - fixes bugs we're affected by
   - brings new feature we are interested in, without breaking the ones we rely on
   - is needed to upgrade another plugin that we want to upgrade
   - is required by a system upgrade (e.g. of the Jenkins packages)
* report bugs identified in Jenkins plugins after they have been upgraded (both
  on the upstream bug tracker and on our own one)
* act as the de facto interface between Tails and the servers hosting
  our services (boum.org, immerda.ch) for non-trivial requests

<a id="tools"></a>

# Tools

The main tools used to manage the Tails infrastructure are:

* [Debian](https://www.debian.org/) GNU/Linux; in the vast majority of
  cases, we run the current stable release
* [Puppet](http://projects.puppetlabs.com/projects/puppet),
  a configuration management system
* [Git](http://git-scm.com/) to host and deploy configuration,
  including our [[Puppet modules|contribute/git#puppet]]

<a id="communication"></a>

# Communication

A few people have write access to the puppetmasters, and can log into
the hosts.  
They read the <tails-sysadmins@boum.org> encrypted mailing list.

We use Redmine tickets for public discussion and tasks management:

* [tasks requiring *Sysadmin*
  work](https://labs.riseup.net/code/projects/tails/issues?query_id=113)
* [tasks belonging to the *Infrastructure*
  category](https://labs.riseup.net/code/projects/tails/issues?query_id=140)

<a id="services"></a>

# Services

## APT repositories

### Custom APT repository

* purpose: host Tails-specific Debian packages
* [[documentation|contribute/APT repository/custom]]
* access: anyone can read, Tails core developers can write
* tools: [[!debpts reprepro]]
* configuration: `tails::reprepro::custom` class
  in [[!tails_gitweb_repo puppet-tails]]

### Time-based snapshots of APT repositories

* purpose: host full snapshots of the upstream APT repositories we
  need, which provides the freezable APT repositories feature needed
  by the Tails development and QA processes
* [[documentation|contribute/APT repository/time-based snapshots]]
* access: anyone can read, release managers have write access
* tools: [[!debpts reprepro]]
* configuration: `tails::reprepro::snapshots::time_based` class
  in [[!tails_gitweb_repo puppet-tails]]

### Tagged snapshots of APT repositories

* purpose: host partial snapshots of the upstream APT repositories we
  need, for historical purposes and compliance with some licenses
* [[documentation|contribute/APT repository/tagged snapshots]]
* access: anyone can read, release managers can create and publish new
  snapshots
* tools: [[!debpts reprepro]]
* configuration: `tails::reprepro::snapshots::tagged` class
  in [[!tails_gitweb_repo puppet-tails]]

## Bitcoind

* purpose: handle the Tails Bitcoin wallet
* access: Tails core developers only
* tools: [[!debpts bitcoind]]
* configuration: `bitcoind` class in [[!tails_gitweb_repo puppet-bitcoind]]

## BitTorrent

* purpose: seed the new ISO image when preparing a release
* [[documentation|contribute/release_process]]
* access: anyone can read, Tails core developers can write
* tools: [[!debpts transmission-daemon]]
* configuration: done by hand ([[!tails_ticket 6926]])

## Gitolite

* purpose: host Git repositories used by the puppetmaster and other
  services; mostly useless for humans
* access: Tails core developers only
* tools: [[!debpts gitolite]]
* configuration: `tails::gitolite` class in [[!tails_gitweb_repo
  puppet-tails]]

## git-annex

* purpose: host the full history of Tails released images and Tor
  Browser tarballs
* access: Tails core developers only
* tools: [[!debpts git-annex]]
* configuration:
  - `tails::git_annex` and `tails::gitolite` classes in
    [[!tails_gitweb_repo puppet-tails]]
  - `tails::git_annex::mirror` defined resource in
    [[!tails_gitweb_repo puppet-tails]]

<a id="icinga2"></a>

## Icinga2

* purpose: Monitor Tails online services and systems.
* access: only Tails core developers can read-only the Icingaweb2 interface,
  sysadmins are RW and receive notifications by email.
* setup: We have one Icinga2 instance installed on a dedicated system
  used as the master of all our Icinga2 zones. We use a VM on the other
  bare-metal host as the Icinga2 satellite of our master. Icinga2 agents are
  installed on every other VM and the host itself. They report back to
  the satellite, which transmits to the master. We spread the Icinga2
  configuration with Puppet. This way, we achieve a certain isolation
  where the master or the satellite have no right to configure agents or
  run arbitrary commands on them.
* tools: [[!debpts icinga2 desc="Icinga2"]], [[!debpts icingaweb2]]
* configuration:
  - master:
    * `tails::monitoring::master` class in [[!tails_gitweb_repo puppet-tails]].
    * some configuration in the ecours.tails.boum.org node manifest.
    * See Vpn section.
  - web server:
    * `tails::monitoring::icingaweb2` class in [[!tails_gitweb_repo puppet-tails]],
       that wraps around [upstream `icingaweb2` module](https://git.icinga.org/puppet-icingaweb2.git).
    * some configuration in the ecours.tails.boum.org node manifest.
  - satellite:
    * `tails::monitoring::satellite` class in [[!tails_gitweb_repo puppet-tails]],
  - agents:
    * `tails::monitoring::agent` class in [[!tails_gitweb_repo puppet-tails]]
* documentation:
  - [[How to add checks to our monitoring setup|roles/sysadmins/adding_icinga2_checks]]

## Jenkins

* purpose: continuous integration, e.g. build Tails ISO images from
  source and run test suites
* access: only Tails core developers can see the Jenkins web interface
  ([[!tails_ticket 6270]]); anyone can [[download the built
  products|contribute/how/testing]]
* tools: [[!debpts jenkins desc="Jenkins"]], [[!debpts jenkins-job-builder]]
* configuration:
  - master:
    * `jenkins` class in [[!tails_gitweb_repo puppet-jenkins]]
    * `tails::jenkins::master` class in [[!tails_gitweb_repo puppet-tails]]
    * a few Jenkins plugins installed with `jenkins::plugin`
    * YAML jobs configuration lives in a
      [[!tails_gitweb_repo jenkins-jobs desc="dedicated Git repository"]];
      [Jenkins Job Builder](http://ci.openstack.org/jenkins-job-builder/)
      uses it to configure Jenkins
  - slaves:
    * `tails::builder`, `tails::jenkins::slave`,
      `tails::jenkins::slave::iso_builder` and `tails::tester` classes in
      [[!tails_gitweb_repo puppet-tails]]
    * some configuration in the manifest ([[!tails_ticket 7106]])
  - web server:
    * some configuration in the manifest ([[!tails_ticket 7107]])

## rsync

* purpose: provide content to the public rsync server, from which all
  HTTP mirrors in turn pull
* access: read-only for those who need it, read-write for Tails core
  developers
* tools: [[!debpts rsync]]
* configuration: `tails::rsync` in [[!tails_gitweb_repo puppet-tails]]

## Tor bridge

* purpose: provide a Tor bridge that Tails contributors can easily use
  for testing
* access: anyone who gets it from
  [BridgeDB](https://bridges.torproject.org/)
* tools: [[!debpts tor]], [[!debpts obfs4proxy]]
* configuration:
  - `tails::apt::repository::torproject` in
    [[!tails_gitweb_repo puppet-tails]]
  - `tor::daemon::relay` in [[!tails_gitweb_repo puppet-tor]]

## VPN

* purpose: flow through VPN traffic the connections between our
  different remote systems. Mainly used by the monitoring service.
* access: private network.
* tools: [[!debpts tinc]]
* configuration:
  - `tails::vpn::instance` class in the [[!tails_gitweb_repo puppet-tails]]
     repo.

## Web server

* purpose: serve web content for any other service that need it
* access: depending on the service
* tools: [[!debpts nginx]]
* configuration:
  - `nginx` class in [[!tails_gitweb_repo puppet-nginx]]
  - hard-coded manifest snippets and files on the puppetmaster
    ([[!tails_ticket 6938]])

## WhisperBack relay

* purpose: forward bug reports sent with WhisperBack to <tails-bugs@boum.org>
* access: public; WhisperBack (and hence, any bug reporter) uses it
* tools: [[!debpts postfix desc="Postfix"]]
* configuration: `tails::whisperback::relay` in [[!tails_gitweb_repo puppet-tails]]
