Postfix + Rspamd do not DKIM-sign forwarded messages [solved!]

[Edited: I have figured a way to deal with this situation; I've written my own solution below, for the benefit of others who came here from Google]

Hi all,

Please take into account that I'm not an expert in Postfix, and have just started playing around with Rspamd (as the alternative to Amavisd & friends). Therefore, I started with Till's tutorial and moved on from there adding a gazillion new options here and there.

Here's the thing: Amavisd & friends just worked — I didn't really care much about what exactly was configured, it seemed to play nice with Postfix (actually, it didn't, but that's not relevant). But, alas, I'm well aware that Rspamd is the way to go, and it seems so much better already, that the last thing I want to do is to go back...

Anyway, here's the issue I've noticed. As part of the insane configurations I've got, there is one kind that stands out from the rest: providing mail forwarding for many domains (if not all), where the users actually only have mailboxes on Gmail (or similar service; but Gmail is preferred by 90% of them), and configure these to have 'mail aliases' with their domain name, hosted on my server.

Sure, I'm aware that there is a lot of back-and-from traffic using such a configuration, but people much rather prefer to trust Gmail than my own server (and I cannot blame them; I do the same...). However, there is a catch: I'm not getting the messages DKIM-signed when they are forwarded out of my server and into Gmail — and, because I have DKIM enabled to reduce forgery attempts and all sorts of Bad Things, Gmail promptly rejects unsigned messages.

Let's imagine the following scenario. We have three users, all with their Gmail mailboxes, [email protected], [email protected], and [email protected]. On ISPConfig, I've created a domain for them — example.com — and have created Email forwards for each of them, e.g.

• [email protected] forwards to [email protected]
• [email protected] forwards to [email protected]
• [email protected] forwards to [email protected]

Deliberately, people internally only send emails to [email protected], [email protected], [email protected], etc. simply because people may change their jobs/functions; more than one person may share the same address (e.g. [email protected] goes to all of them); and sometimes external consultants do not want others to know their Gmail address but only their [email protected] address. Last but not least, it's always much more reasonable to print your business card with marketing@example com as opposed to [email protected].

So imagine that the boss now wants to send emails to his other two employees. He logs into Gmail using [email protected], and, having properly configured Gmail (aye, that step has been checked over and over again...), sends emails to [email protected] and [email protected].
From the perspective of Gmail, these two addresses are hosted 'somewhere' on the Internet. All it knows is that these are served by the MX record of example.com, which is obviously my.server.tld. Gmail checks if everything is ok with my.server.tld, signs the messages with DKIM (and ARC), and sends them to my.server.tld to be processed by Postfix.

So far, so good, this is how it has always worked since at least 2007, having gone through several providers, different servers, and most ISPConfig versions until today.

On my.server.tld, Postfix does its checking. It validates the origin — Gmail — and looks at the SPF records: yep, Gmail is allowed to send mail on behalf of example.com, too. It self-validates itself as the correct MX record, checks that Google hasn't been put again on a blacklist (happens much more frequently than I thought!), and launches Rspamd, which will do further checks — for spam, for antivirus, and, of course, to check if Gmail did properly DKIM-sign everything.

Everything is A-OK, so it's time to do local delivery. Postfix loads up its aliases (well, retrieves them from MySQL, I guess) and finds out that a, b and c are not local mailboxes. Instead, all email to them is to be forwarded without delay to... Gmail again. All checks are being run as expected (e.g. to see if we're not accidentally sending spam....), but... Rspamd 'forgets' to DKIM-sign the messages! (it doesn't ARC-sign them, either).

Well, as expected, Gmail receives those messages. They are for valid Gmail mailboxes, but... example.com mandates that all email from that domain has to be properly DKIM-signed. Which these messages are not. So, Gmail bounces them back with the usual error:
550-5.7.26 This message does not have authentication information or fails to pass authentication checks. To best protect our users from spam, the message has been blocked.

Now, this is what I expect Gmail to do if they receive a message 'pretending' to come from example.com; so this is the expected behaviour. I've checked: under this particular scenario, Rspamd does not DKIM-sign the messages!

To spare you the usual questions...

• Yes, I have tested forwarding to other providers, not only Gmail. The behaviour is exactly the same.
• Yes, I have tested sending messages from Roundcube running on my.server.tld. These get signed without any problem.
• Yes, I have tested sending messages from Gmail using a local, authenticated account on my.server.tld, to a completely different account (no relation to example.com or any other domain hosted by me). All these get properly DKIM-signed by Rspamd running on my.server.tld.
• Yes, I have configured a local client (instead of using Gmail's web interface), namely Apple Mail, and repeated the above tests a gazillion times. The result was always the same: if the origin and destination are Gmail mailboxes, but they were forwarded through my.server.tld, they never get DKIM-signed (but pass all other checks).
• Yes, this also happens with other domain names hosted on my server with similar setups (i.e. it's not a mistake done only with example com; rather, it's a systematic error I made somewhere and which affects everything).

In short, what I need is to figure out what rule, either on Rspamd or Postfix, is responsible for activating DKIM-signing when an email is being forwarded through my server, but it's neither the origin nor the destination. In other words, in this particular scenario — and only that one! — my.server.tld is acting as a very weirdly configured relaying service, which, to work properly, needs to DKIM-sign (these days, even ARC-sign!) all outgoing emails even if they haven't originated on my server.

My solution!

Browsing through the many entries that Google and Bing find for this issue, it's clear that most people out there believe that the behaviour described above is built-in into Postfix/Rspamd; in other words, Postfix + Rspamd have no way of authenticating forwarded messages!

There seem even to be a few scattered suggestions on why this is the case: the more popular answer is that 'mail forwarding' is a moribund technology, and that all services building on top of that ought to be discontinued/removed. In other words, not being able to forward email is the intended behaviour, in these troubled days where spammers use all tricks of the trade to get their spam through — and using multiple forwarders to obfuscate the actual source is a very common method.

On the other hand, there are at least two (possibly more!) perfectly legitimate scenarios for enabling forwarding and properly signing it:

1. Mailing lists (especially those that are real mailing lists, not so-called 'marketing newsletters' which are only a more legitimate form of spam).
2. My own scenario, i.e. allowing people to use their preferred mail service hosted externally, but have legitimate email aliases that are linked to a domain address hosted elsewhere.

To allow those, there is a little trick — and a woefully undocumented one (probably for good reasons), which I found by sheer chance, after browsing dozens of tutorials and explanations. It's a minor component of Postfix called postsrsd. SRS stands for 'Sender Rewriting Scheme' (yay, another acronym to learn!) and can be simply described as follows:

When dealing with forwarded email which got DKIM-signed along an ARC chain, there is a catch: at some point, our server will need to add a signature to validate the original message coming from the remote server. We can certainly check that what we've received is ok via DKIM, because we access the public key from that remote server. However, there is no way to sign the message again when forwarding it, since obviously we don't have access to the private key for signing it! That's, in a nutshell, why Postfix + Rspamd cannot sign forwarded emails: they obviously have no way to sign a message that did not originate on the server.

So, what does SRS do? Essentially, it rewrites the message, keeping the content intact, but changing the headers to (legitimately) pretend that it had originated on our server. To do that, it generates a pseudo-alias (linked to the local domain), using locally stored cryptographic keys, but keeping the original email address as well, and stores the generated alias on an internal lookup table. The message is then forwarded under the new email address in its headers — properly signed, not breaking the ARC chain, since it gets signed with the DKIM private key stored on our server.

When the message is delivered, the remote MTA looks at the headers, and sees that the origin of the email is our server (and not the original one!). It then asks our server if that strange-looking email address belongs to a valid, authenticated user. Our server doesn't recognise the pseudo-alias, so it asks postsrsd if it happens to know that specially-crafted pseudo-address. postsrsd breaks it up, finds the original email address, looks up on its internal database if that email address has been seen before and has a valid cryptographic signature. Note that our server doesn't really check if that email address is valid by itself; after all, Postfix + Rspamd had already verified that when the message showed up for the first time. postsrsd just sort of 'believes' that Postfix + Rspamd have already done their job, and just affixes its own special signing to that address — essentially saying, 'yep, that's a valid message we've received before, and I recognise it, because the small cryptographic signature I have embedded in the pseudo-alias is a valid one'.

postsrsd tells Postifx that it recognises the address, so Postfix talks back to the remote server and says, 'yep, that email address may look weird, but it's a valid, authenticated user that we know in our system'. The remote MTA accepts the message, does some rewriting magic, and voilà, the message gets delivered with an intact ARC chain, and the From: header will actually match the one from the original message.

As you can imagine, a badly configured system may falsely accept a forged address. That's why spammers have recently gone crazy with this SRS method of 'obfuscating' the email address. You might have seen in your logs spam coming from addresses such as <[email protected]>. What the spammers are trying to accomplish is that SRS-aware servers will look at that pseudo-alias and 'understand' that it might be a valid, SRS-generated address. Indeed, if anyone in the middle of the chain looks at that address, it can see that there is this embedded [email protected] address inside it; this is my valid email address, and mail servers can ask my server if that user exists and is a valid one — which is true. As such, many servers may just assume that this is a legitimately forwarded message from a mailing list and simply let it go through.

They are also hoping that the final destination hasn't got all their anti-spam mechanisms finely tuned. When properly configured, our server will not get 'tricked' by this. You see, the part that says 'noel-rios' is supposed to be a digital signature; but of course the spammers have no way to guess what the actual private key is (since it's only stored on my server), so they just generate something randomly. Because no other server can validate that short signature except my own server, under-configured servers, knowing only that this may be a legitimately forwarded message, can pass it along to the final destination...

Obviously, that specific message will never be delivered on my server, since postsrsd will check for a valid signature — and immediately detect a forgery. Of course, in the real world, many other anti-spam features would have already dismissed the message as coming from a spamming location (peletonfraud.com in the above example), and pretty much ignore the fake address and its format. But spammers rely always on slightly misconfigured or incorrectly configured MTAs that will happily forward some of their messages, and hope that at least a few will reach their intended destination. 'Pretending' that a spam message actually comes from what looks like a legitimate mailing list is one of the latest trends in spamming technology. But even a brand-new domain name configured in a freshly installed server (which technically hasn't any 'bad reputation' yet on the many blacklists out there) will be unable to trick postsrsd into accepting a forged message as valid. While in theory such a message may get bounced around by several mail-obfuscating and -forwarding gateways, unsuspectingly passing along what looks like a legitimate email, it will never reach its destination.

postsrsd has a minimalistic setup and can be deployed in minutes (there are some extra options and plugins; take a look at the GitHub page). It's available via APT (under Debian/Ubuntu) and very likely many other repos as well. It is almost mandatory if you're running public mailing lists if any of its members is not a local user. One might wonder why postsrsd is not installed by default with Postfix, or, at least, why it's not mentioned in the vast majority of tutorials (including those here at HowToForge!).

Well, the main reason that I can think of is that most mail servers will not act as mail forwarders in any situation, and, in fact, because being a mail forwarder is a potential liability (configure it wrongly, and it can be attacked by spammers which will use it as their mail forwarder....), and, as far as I understand, there are really just two use-case scenarios for it. Not many servers will host mailing lists (it's that kind of technology that is declining); perhaps even fewer will be interested in mail forwarding, especially because so many mail services have the option to import email from remote servers via POP3 or IMAP4 — it's much more safer to create a local mailbox and set up Gmail (or anything else) just to retrieve email from it regularly. That avoids all troubles with mail forwarding!

Unfortunately, there are still very good and legitimate reasons to do 'old-school' mail forwarding — the main one being that people are not that computer-savvy. Giving them detailed instructions on how to configure Gmail to regularly retrieve email from a remote mailbox, just so in order to allow you to send email with that address as a Gmail alias, may be asking too much of the vast majority of users. After all, one of the biggest 'selling points' of Gmail is that it's just a web page with which you interact, with zero configuration. Everybody loves webmail — because it's universally available on any device and accessible from anywhere you've got network coverage, without the need for any kind of special configuration: you just point a browser to gmail.com, and you're done.

For these kinds of users (and these will be the majority!), being able to forward email on their behalf, without requiring them to change anything on their configuration, is a blessing. After all, that's how it has been done since times immemorial — it's just in recent years that email configurations have become insanely complex in order to continue the eternal war against spam...

Whew. Feel free to correct anything I've written and, if you think that this may be useful to explain in your own tutorials, go ahead and copy & paste whatever seems interesting.

Note that you'll have to be extra-careful with your settings change, since they may be partially overwritten by ISPConfig in upcoming releases, or if you wish to force a reconfiguration (because you've activated new features, for instance). My own configuration is a complex mess because of that (ISPConfig3 is not to blame, mind you!!), because I have to take that into account, and that's why I'm not reproducing it — just broadly describing how it works.

From my point of view, my issue is 'solved' (although, obviously, I'll gladly accept any suggestions for improvements!), and I'm mostly posting this here because I'm well aware that Google so often returns searches with links to HowToForge — just because the quality of the information contained here is so high (if it weren't, and I hadn't lurked in the forums in search of issues so many years ago, I would not even been aware of the existence of ISPConfig3...).

Cheers all! Happy Postfix hacking!

 

postsrsd is the exact solution for this. I've been using it for a couple of years now and it's the ideal solution for both your problem and, theoretically, mailing lists. Glad you found it.

 

in your case there is also an alternative.
if the boss of example.com is logging in to gmail as eg [email protected], to send mail to [email protected] and [email protected], he doesn't have to send the mail as [email protected].
[email protected] has their account configured, so under settings-accounts and import-send mail as, the [email protected] address is configured, and not as an alias. this should then require server & auth details for a mail account on your ispconfig mailserver.
you can do this for all the @example.com email addresses in their gmail accounts.
you just need to create a different send-only email address on your mailserver for them to send through. this way they send the mail out as *@example.com, which is directly authenticated and dkim signed by your mailserver. gmail has nothing to complain about when the mail is forwarded to the destination gmail accounts.
to avoid having to create lots of different send-only mailboxes, you can even have them all use the same single send-only mailbox, although of course this has other complications if that mailbox account gets compromised.


but yes, postsrsd is a nice simple solution, it's just a shame it changes everything to use the mailservers main domain as the new source, a spammer going through a clients mailforward can ruin your own domain's reputation. would be nice if the source domain it changes the mailforward from address to was the actual forwarding domains domain name.