seppo/doc/00016_--_arch_design.txt

Q: what about qeueuing?

A: yes

Several operations may finish slowly, so they have to be queued. At the heart
there is a wait job queue with one file for each job still due.

app/var/spool/jobs/wait/2023-03-13T012345Z.5abf8.12.job

The date is the time until which th job is postponed, hash is over the unique job properties, the number is
how many tries there have been before. The file content is immutable.

Retries double the waiting time each time and timeout after one week. Starting
with a 1-minute wait that's 13 tries.

Example calculation: Advertise a post to 10k followers - doing 10k sequential
jobs each taking 2s is 20ks or 5.5h.

The queue health (it should be all empty) is displayed as part of the housekeeping
Web UI. File counts, time slots for wait.

Queue processing is triggered by

- loop until no more job in new
- sleeping forks (until next one)
- logged in web interaction
- endpoint with auth token (monitoring)

There should not be more than 1 sleeping fork, however => app/var/lock/job

Only jobs entering err log an INFO.

Q: How to ensure loss-safe retries? How/where to store retry counter?
   http://cr.yp.to/proto/maildir.html
   - in the filename? duplicate jobs possible
   - in the job? otherwise unchanged payload rewrites and timing issues.

   API evtl. https://salmon-mail.readthedocs.io/en/latest/_modules/salmon/queue.html