Outages

About this page

Occasionally you'll see this fellow on your visit to our site. Whenever this happens we've made a mistake. This page is here to help you understand what went wrong and (hopefully) give you a sense that we're constantly improving. If you find any problems using the site please report a bug here

We hope you won't have to refer to this page too often. :)

2016-08-31

What happened

NotABug.org was completey unreachable for most of the 31st of August and the 1st of September. The frontend was not reachable, git pulls and pushes were not possible.

What caused it

Due to a bad DIMM and non-ECC RAM in the server the filesystem of the main gogs repository got corrupted. This had to be restored from backup. No data was lost, however

How has this been addressed

A new server with ECC ram was purchased.
All data was succesfully restored from backups.

2016-01-31

What happened

NotABug.org was completey unreachable for most of the 31st of January and the 1st of February. The frontend was not reachable, git pulls and pushes were not possible.

What caused it

Due to a drive failure in the array the filesystem housing the Git repositories went in read-only mode. During recovery of the situation a admin-error accidentally deleted the root filesystems of both the hypervisor and the NotABug.org VM. Due to this error the SSH key of the server was changed. The new fingerprints are:

1024 SHA256:3Gfd/hCr2FAnV34U/LKO93m14yD7JnJOfZPub2sK7No (DSA)
256 SHA256:tv7X1j2jOYPgztNC+3KYh2E751I01Y7vHWS83uax8jQ (ECDSA)
256 SHA256:BtR8pJSuIPDCUT5DJ0kk/Sxspp9dgWf5p3EE0OUQe3g (ED25519)
2048 SHA256:C3P4FLq03YMBgA0ipZrw8AvYLQsq54I1lYUPhPKI5iE (RSA)
1024 MD5:c3:27:ea:d8:08:52:f7:c5:5a:68:4d:37:c3:50:32:35 (DSA)
256 MD5:44:1a:ae:46:fe:c9:a7:f9:99:77:21:56:bc:02:47:de (ECDSA)
256 MD5:fe:ec:8b:51:90:05:bc:3f:dd:21:94:31:46:12:72:d6 (ED25519)
2048 MD5:b2:de:dc:04:31:c4:18:85:10:ee:c9:d7:e4:60:57:7d (RSA)

Additionally, all the avatars were lost. No repository data was lost, however.

How has this been addressed

The systems were reinstalled and restored from backup where possible.

2015-07-18

What happened

NotABug.org web front-end was throwing 500 errors for most of the day. It seems git pushes also did not work.

What caused it

Someone has flooded the Gogs install with thousands of new user registrations. We were effectively suffering a denial of service attack.

How has this been addressed

The Gogs VM was restarted.
The new users were deleted.

More work still needs to be done to make this type of attack less likely to succeed in the future. No concrete steps have been taken in this direction, however.

2015-04-20

What happened

NotABug.org web front-end was down between 19:00 and 19:30 CEST. It was impossible to view any of the NotABug.org web content. Git pulls and pushes were unaffected.

What caused it

The Gogs software crashed and the monitoring software responsible for restarting it automatically was still faulty. We now know that the problem was that the gogs daemon expects to be able to read the current userid from the $USER environment variable. This only works if the init script is started with the shell runing in bash mode. The init script was called with /bin/sh when running non-interactively (during startup and during the monitor cron job).

How has this been addressed

The init script has been adjusted such that it explicitly sets the $USER variable.

2015-04-17

What happened

NotABug.org web front-end was down between 17:00 and 19:00 CEST. It was impossible to view any of the NotABug.org web content. Git pulls and pushes were unaffected.

What caused it

The Gogs software crashed and the monitoring software responsible for restarting it automatically was faulty causing a restart loop.

How has this been addressed

The monitoring software has been adjusted such that a restart loop should be less likely.
Investigations are ongoing to find the root cause of the crash.

2015-03-12

What happened

NotABug.org web front-end was down between 17:00 and 18:00 CEST. It was impossible to view any of the NotABug.org web content. Git pulls and pushes were unaffected.

What caused it

We currently believe that a bad merge of one of Gogs' dependencies caused a problem with the application. At the time the only person who could fix the problem was off at a party. There was no monitoring. Additionally the error page offered no recourse making it seem like we had just gone away.

How has this been addressed

The dependencies of Gogs were reverted to known-good versions.
More people were given access to troubleshoot the Gogs instance for NotABug.org.
Basic monitoring was put in place which will be improved in the future.
A better error page was created with contact information.