Migrating from procmail to sieve

pdo debian eng

Anarcat's "procmail considered harmful" post convinced me to get my act together and finally migrate my venerable procmail based setup to sieve.

My setup was nontrivial, so I migrated with an intermediate step in which sieve scripts would by default pipe everything to procmail, which allowed me to slowly move rules from procmailrc to sieve until nothing remained in procmailrc.

Here's what I did.

Literature review

https://brokkr.net/2019/10/31/lets-do-dovecot-slowly-and-properly-part-3-lmtp/ has a guide quite aligned with current Debian, and could be a starting point to get an idea of the work to do.

https://wiki.dovecot.org/HowTo/PostfixDovecotLMTP is way more terse, but more aligned with my intentions. Reading the former helped me in understanding the latter.

https://datatracker.ietf.org/doc/html/rfc5228 has the full Sieve syntax.

https://doc.dovecot.org/configuration_manual/sieve/pigeonhole_sieve_interpreter/ has the list of Sieve features supported by Dovecot.

https://doc.dovecot.org/settings/pigeonhole/ has the reference on Dovecot's sieve implementation.

https://raw.githubusercontent.com/dovecot/pigeonhole/master/doc/rfc/spec-bosch-sieve-extprograms.txt is the hard to find full reference for the functions introduced by the extprograms plugin.

Debugging tools:

doveconf to dump dovecot's configuration to see if what it understands matches what I mean
sieve-test parses sieve scripts: sieve-test file.sieve /dev/null is a quick and dirty syntax check

Backup of all mails processed

One thing I did with procmail was to generate a monthly mailbox with all incoming email, with something like this:

BACKUP="/srv/backupts/test-`date +%Y-%m-d`.mbox"

:0c
$BACKUP

I did not find an obvious way in sieve to create montly mailboxes, so I redesigned that system using Postfix's always_bcc feature, piping everything to an archive user.

I'll then recreate the monthly archiving using a chewmail script that I can simply run via cron.

Configure dovecot

apt install dovecot-sieve dovecot-lmtpd

I added this to the local dovecot configuration:

service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    user = postfix
    group = postfix
    mode = 0666
  }
}

protocol lmtp {
  mail_plugins = $mail_plugins sieve
}

plugin {
  sieve = file:~/.sieve;active=~/.dovecot.sieve
}

This makes Dovecot ready to receive mail from Postfix via a lmtp unix socket created in Postfix's private chroot.

It also activates the sieve plugin, and uses ~/.sieve as a sieve script.

The script can be a file or a directory; if it is a directory, ~/.dovecot.sieve will be a symlink pointing to the .sieve file to run.

This is a feature I'm not yet using, but if one day I want to try enabling UIs to edit sieve scripts, that part is ready.

Delegate to procmail

To make sieve scripts that delegate to procmail, I enabled the sieve_extprograms plugin:

 plugin {
   sieve = file:~/.sieve;active=~/.dovecot.sieve
+  sieve_plugins = sieve_extprograms
+  sieve_extensions +vnd.dovecot.pipe
+  sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve-pipe
+  sieve_trace_dir = ~/.sieve-trace
+  sieve_trace_level = matching
+  sieve_trace_debug = yes
 }

and then created a script for it:

mkdir -p /usr/local/lib/dovecot/sieve-pipe/
(echo "#!/bin/sh'; echo "exec /usr/bin/procmail") > /usr/local/lib/dovecot/sieve-pipe/procmail
chmod 0755 /usr/local/lib/dovecot/sieve-pipe/procmail

And I can have a sieve script that delegates processing to procmail:

require "vnd.dovecot.pipe";

pipe "procmail";

Activate the postfix side

These changes switched local delivery over to Dovecot:

--- a/roles/mailserver/templates/dovecot.conf
+++ b/roles/mailserver/templates/dovecot.conf
@@ -25,6 +25,8 @@
…
+auth_username_format = %Ln
+
…
diff --git a/roles/mailserver/templates/main.cf b/roles/mailserver/templates/main.cf
index d2c515a..d35537c 100644
--- a/roles/mailserver/templates/main.cf
+++ b/roles/mailserver/templates/main.cf
@@ -64,8 +64,7 @@ virtual_alias_domains =
…
-mailbox_command = procmail -a "$EXTENSION"
-mailbox_size_limit = 0
+mailbox_transport = lmtp:unix:private/dovecot-lmtp
…

Without auth_username_format = %Ln dovecot won't be able to understand usernames sent by postfix in my specific setup.

Moving rules over to sieve

This is mostly straightforward, with the luxury of being able to do it a bit at a time.

The last tricky bit was how to call spamc from sieve, as in some situations I reduce system load by running the spamfilter only on a prefiltered selection of incoming emails.

For this I enabled the filter directive in sieve:

 plugin {
   sieve = file:~/.sieve;active=~/.dovecot.sieve
   sieve_plugins = sieve_extprograms
-  sieve_extensions +vnd.dovecot.pipe
+  sieve_extensions +vnd.dovecot.pipe +vnd.dovecot.filter
   sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve-pipe
+  sieve_filter_bin_dir = /usr/local/lib/dovecot/sieve-filter
   sieve_trace_dir = ~/.sieve-trace
   sieve_trace_level = matching
   sieve_trace_debug = yes
 }

Then I created a filter script:

mkdir -p /usr/local/lib/dovecot/sieve-filter/"
(echo "#!/bin/sh'; echo "exec /usr/bin/spamc") > /usr/local/lib/dovecot/sieve-filter/spamc
chmod 0755 /usr/local/lib/dovecot/sieve-filter/spamc

And now what was previously:

:0 fw
| /usr/bin/spamc

:0
* ^X-Spam-Status: Yes
.spam/

Can become:

require "vnd.dovecot.filter";
require "fileinto";

filter "spamc";

if header :contains "x-spam-level" "**************" {
    discard;
} elsif header :matches "X-Spam-Status" "Yes,*" {
    fileinto "spam";
}

Updates

Ansgar mentioned that it's possible to replicate the monthly mailbox using the variables and date extensions, with a hacky trick from the extensions' RFC:

require "date"
require "variables"

if currentdate :matches "month" "*" { set "month" "${1}"; }
if currentdate :matches "year" "*" { set "year" "${1}"; }

fileinto :create "${month}-${year}";