forgefed

ForgeFed extension for Pagure

This software is an extension for the Pagure forge that adds ForgeFed federation to Pagure instances.

This work has been made possible by the help of, and with the financial support from, NLNet and the NGI0 Discovery Fund.

Requirements

• Python >= 3.7 (the Python re module has replaced re._pattern_type with re.Pattern starting with Python 3.7)

Installation

This plugin must be installed in a working instance of Pagure. See Installing Pagure for how to install it. The plugin reads most of the settings from the Pagure configuration file. Make sure that it is configured and that the instance is running correctly before installing the plugin.

1. Install the Python dependencies, either globally or in the Pagure virtual environment (if using any):

   pip install -r requirements.txt
   

2. Now we need to configure Pagure to load the plugin. Pagure uses two configuration files: one is the main configuration for the app, the other one is for loading plugins. Start by setting the variable PAGURE_PLUGINS_CONFIG in the pagure main config file. Then edit the secondary config and import the forgefed plugin like this:

   PLUGINS_PATH = "/path/to/plugins/folder"
   if PLUGINS_PATH not in sys.path:
       sys.path.append(PLUGINS_PATH)
   import forgefed.app
   PLUGINS = [ forgefed.app.APP ]
   

3. We need to register a signal listener for listening to Pagure events. Add this to the Pagure config file:

   import blinker
   def notify_forgefed(sender, topic, message):
       import forgefed.tasks
       forgefed.tasks.notification.handle_pagure_signal.delay(
           topic, message, os.environ.get('FORGEFED_WORKER'))
   if not blinker.signal('pagure').has_receivers_for(notify_forgefed):
       blinker.signal('pagure').connect(notify_forgefed, weak=False)
   

4. Configure Celery imports in the Pagure main config file, such that all the tasks are found automatically at startup:

   # Worker configuration
   import sys
   PLUGINS_PATH = "/home/git/pagure_local/plugins"
   if PLUGINS_PATH not in sys.path: sys.path.append(PLUGINS_PATH)
   CELERY_CONFIG = {
       'imports': ('pagure.lib.tasks', 'forgefed.tasks'),
       'task_annotations': {
           '*': {
               'autoretry_for':     [ Exception ],
               'retry_kwargs':      {
                   'max_retries': 20
               },
               'retry_backoff':     True,     # Exponential backoff
               'retry_backoff_max': 60*60*24, # Do not delay for more than 24h
               'retry_jitter':      True,
               #'rate_limit':       '10/s'
           }
       }
   }
   

5. Pagure uses Celery with several queues for handling its tasks asynchronously (eg. for creating a new repo, sending emails, or mirroring a repo). We have to start a new Celery worker for handling the forgefed queue:

   ./start_forgefed_worker.sh
   

Testing

• After the plugin is installed, it's sufficient to start Pagure along with the ForgeFed worker that will handle federation requests asynchronously
• Celery workers do not reload automatically if the tasks file has been modified. It's necessary to restart the worker manually.
• To run the test suite, copy the tests_forgefed folder into the pagure directory (the same where there is also the Pagure's own test folder) and run pytest tests_forgefed. This is required because the test suite needs to load Pagure and its modules.
• Two instances are available for testing: pagure1, pagure2

Links

• ForgeFed spec and documentation
• NLNet
• NGI
• NGI0 Discovery Fund
• Peers

License

This is free software licensed as GNU General Public License, version 2. See the license files.

SPDX-FileCopyrightText: 2020-2021 zPlus zplus@peers.community SPDX-License-Identifier: GPL-2.0-only