debian_pkg-listmaster/bits/sept_2007_bits

From: whoever sends this
Subject: Bits from the listmasters

Hi everyone,

As you might have already read from several posts on debian-devel or
debian-user, lists.debian.org has a new spam filter setup. This was
done during the listmaster@ + owner@bugs meeting in Essen, Germany.
But to review everything in a single message:

Internals
---------

* Amavis-Setup
  The new spam filter setup of lists.debian.org includes the use of
  amavisd-new. We are using a feature called policy-banks, where we
  have grouped all 180 mailing lists into the following policy banks
  plus a few more administrative ones:

    * bug               * lang-greek
    * en-ht	        * lang-hu-fi
    * en-lt	        * lang-indic  
    * lang-arabic       * lang-indonesic
    * lang-asian        * lang-romanic
    * lang-esperanto    * lang-scandinavic
    * lang-french       * lang-slavic
    * lang-germanic     

  Each policy bank has its own spam filtering setup. Most of it can be
  looked at, as it is checked into svn[1]. To find out to which policy
  bank a list belongs, look for the X-Virus-Scanned header in the email.

  The advantage of this new setup is that now we can distinguse between
  different list types, and can set filters and scorings for each
  list type on its own. 

  Mails to each list can be "ham", "maybe-spam" and "spam". For
  borderline messages (maybe-spam) we are currently implementing a
  queueing mechanism, which allows us to delay these mails for a while
  and on recheck them after a defined time has passed.

  Gandalf
  ~~~~~~~
  Don Amstrong is currently implementing a new greylisting daemon we
  want to use on lists.debian.org. You now might ask, why another
  greylisting daemon? We were inspired by the sort of postfix-weight is
  working, but think it has some design flaws. Also we consider some
  reporting mechanism back from spamassassin back into the greylisting
  daemon quite helpful. Stay tuned, as we want this feature going live
  rather soon.

  Considering of lurker as webfrontend
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  The listmaster team currently evaluating lurker as official
  additional webfrontend for the mailing list archive. A few
  show-stoppers have been found and documented in [2]. These have been
  forwarded to the lurker upstream who is also Debian developer. We
  hope to have these changes implemented rather soon, so we can also
  offer lurker as an official web archive.

  SVN on Alioth
  ~~~~~~~~~~~~~
  We have moved some non-confidential configuration files to an SVN
  repository on Alioth, including our SpamAssassin and amavisd-new
  configuration. It can be viewed here[1]. If you want to help us with
  spam filtering, see if you can improve the SpamAssassin rule files.
  Send patches to them to listmaster@lists.debian.org.

  Team members
  ~~~~~~~~~~~~
  There has been quite a bit of restructuring of listmaster team
  members, since we last sent out an official bits from the
  listmasters. New members were added, some old members left the team.
  All of them did tremendous good work as listmasters and we want to
  thank them. Our thanks goes to Jaakko Niemi, Anand Kumria, Frans
  Pop, Robert McQueen and Cesar Mendoza.

  Recently we have also added three more team members, Don Armstrong,
  David Moreno Garza and Thomas Viehmann, last one doing listarchives
  only.

  Clean up
  ~~~~~~~~
  We use smartlist[3] for running the lists. Smartlist consists of a
  series of C programs, procmail and shellscripts. The setup was
  deployed originally sometime in 1998 (judging from some file
  timestamps) and since then it has evolved. Currently we have 180
  lists and each of it has more than 30 files that define how it works
  (maxsize, moderation, ...). That sums up to more than 6000 files we
  have to maintain.

  So now we have been cleaning up and linking identical files
  together, reducing the differing configuration files to ideally one
  file per list.

  During the listmaster meeting this progress started, and about 1000
  files have been linked together. The process of simplifying and
  unifying configurations is still in process.

  There are also some spam filter remnants in these configurations,
  that are also being moved into the spamassassin-config.

  whitelist
  ~~~~~~~~~
  While it is possible to post with an address which isn't subscribed
  to the lists, we recommend that you subscribe to our white-list
  (http://lists.debian.org/whitelist/) so that our system recognizes
  you. This will reduce the risk of false positives causing your mail
  to be dropped.

  Cooperation between bugs and lists
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  During the meeting in Essen Don Amstrong and Robert Blarson from
  bugs.debian.org team were present. This helped quite a lot, as we were
  able to ease the configuration of the spamfilters on both sides, so we
  are now using mostly the same SpamAssassin config, which should
  improve the spam situation on both sides.

How to help listmasters against spam
------------------------------------

   * If you notice a spam in the list archives, press the 'Report As
     Spam'-Button.
   
   * If you run some spam-protecting mechanisms like
     + greylisting
     + tdma (challenge-response system)
     + virus scanner
     + spamscanner
   
     make sure that it doesn't reject mails from murphy.debian.org (and
     master.debian.org), as our bounce detection software is likely to
     unsubscribe you.  From the figures above you can see that we filter
     a lot of spam and malware, but as we run a
     posting-is-open-for-everyone policy, there will always new kinds of
     junk that will pass our filters.
   
   * Do not ever report spam received through our lists to third parties
     services. They are likely to blacklist us or complain to our ISP,
     both of which result in degraded performance for yourself and
     others. This is also likely to cause tension between us, our
     sponsors and their ISPs.
   
   * Report spam that gets to you through our filters to
     report-listspam@lists.debian.org. Please leave all the headers
     untouched. The best method is to bounce (as in mutt) them. There is a
     plugin for thunder^Wicesomething to do that at
     http://mailredirect.mozdev.org/ . DO NOT do this automagically. If
     you want to help us, you must make personally sure that the things
     you report are REALLY spam.
   
   * If you receive lots of spam and know how to stop it through
     procmail or spamassassin, send us (listmaster@lists.debian.org) a
     note with the recipe, or contact us in OFTC #debian-lists
   
   * If you really want to use some kind of auto-responder, make sure
     that it is sane, and interprets the Lists and Precedence-Headers
     correctly so it ignores our mails. If we find that your mail
     address issues automatic responses to the list or subscribers,
     we'll unsubscribe you from all lists.
   
   * Don't subscribe to our lists with a forwarding mail address, if something
     goes wrong with the mail address you are forwarding to, it will be harder
     for us to find out exactly which address we have to drop. Instead, please
     subscribe with the address on which you will be reading the mail. You are
     free to send responses with another address, so your receiving
     address isn't published.

   * Please keep in mind that the Mails to our public lists are
     publically archived at lists.debian.org and many other services
     on the net. This means that everything in your mail is public,
     including your sending mail address.

[1] http://svn.debian.org/wsvn/pkg-listmaster/
[2] http://wiki.debian.org/Teams/ListMaster/LurkerProblems
[3] http://packages.debian.org/smartlist

# vim: tw=72