DKIM for locally generated mail isn't working (Debian11, Ispconfig 3.2, Amavisd-new)

I'm operating a Debian11 server with Ispconfig 3.2. DKIM signing is done by amavisd-new-2.11.1 (20181009).
I've noticed that locally generated outgoing mail (directly with the php mail() function from the domain's associated website) is not signed by DKIM. It works when it's sent through SMTP (connecting to localhost:25, unauthenticated) and verifies ok so DKIM is set up properly, keys work, etc.
The return-path and from headers are OK in both cases (i.e. mail is coming from webmaster@domain as they should).
What am I missing to fix this? I recall that there was an earlier one liner patch to amavisd-new to fix some sort of local DKIM signing issue but I'm not sure if it's relevant.

 

There are multiple ways to achieve this. The Issue is that locally generated mails are not getting handed over to amavis.
One way could be to change the master.cf to something like this:

Code:

pickup  fifo    n       -       n       60      1       pickup
  -o content_filter=  // COMMENT THIS OUT
  -o receive_override_options=

When you comment the line out it should work

 

Thank you for the quick response! Unfortunately that's not the case, the pickup line (ha ha) has no extra options and amavisd does scan it, i.e. the non dkim-signed local message has an "X-Virus-Scanned: Debian amavisd-new at hostname" header but no DKIM.

 

Oh well that is a different issue than. Can you post your master.cf and amavis config(i assume it is called 50_user.conf or something similar)?

 

Here they are - it's mostly stock except for a small change in master.cf to add policy-spf

master.cf

Code:

#
# Postfix master process configuration file.  For details on the format
# of the file, see the master(5) manual page (command: "man 5 master" or
# on-line: http://www.postfix.org/master.5.html).
#
# Do not forget to execute "postfix reload" after editing this file.
#
# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (no)    (never) (100)
# ==========================================================================
smtp      inet  n       -       y       -       -       smtpd
#smtp      inet  n       -       y       -       1       postscreen
#smtpd     pass  -       -       y       -       -       smtpd
#dnsblog   unix  -       -       y       -       0       dnsblog
#tlsproxy  unix  -       -       y       -       0       tlsproxy
submission inet n       -       y       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
#  -o smtpd_tls_auth_only=yes
#  -o smtpd_reject_unlisted_recipient=no
#  -o smtpd_client_restrictions=$mua_client_restrictions
#  -o smtpd_helo_restrictions=$mua_helo_restrictions
#  -o smtpd_sender_restrictions=$mua_sender_restrictions
#  -o smtpd_recipient_restrictions=
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
#  -o milter_macro_daemon_name=ORIGINATING
smtps     inet  n       -       y       -       -       smtpd
  -o syslog_name=postfix/smtps
  -o smtpd_tls_wrappermode=yes
  -o smtpd_sasl_auth_enable=yes
#  -o smtpd_reject_unlisted_recipient=no
#  -o smtpd_client_restrictions=$mua_client_restrictions
#  -o smtpd_helo_restrictions=$mua_helo_restrictions
#  -o smtpd_sender_restrictions=$mua_sender_restrictions
#  -o smtpd_recipient_restrictions=
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
#  -o milter_macro_daemon_name=ORIGINATING
#628       inet  n       -       y       -       -       qmqpd
pickup    unix  n       -       y       60      1       pickup
cleanup   unix  n       -       y       -       0       cleanup
qmgr      unix  n       -       n       300     1       qmgr
#qmgr     unix  n       -       n       300     1       oqmgr
tlsmgr    unix  -       -       y       1000?   1       tlsmgr
rewrite   unix  -       -       y       -       -       trivial-rewrite
bounce    unix  -       -       y       -       0       bounce
defer     unix  -       -       y       -       0       bounce
trace     unix  -       -       y       -       0       bounce
verify    unix  -       -       y       -       1       verify
flush     unix  n       -       y       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
proxywrite unix -       -       n       -       1       proxymap
smtp      unix  -       -       y       -       -       smtp
relay     unix  -       -       y       -       -       smtp
        -o syslog_name=postfix/$service_name
#       -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
showq     unix  n       -       y       -       -       showq
error     unix  -       -       y       -       -       error
retry     unix  -       -       y       -       -       error
discard   unix  -       -       y       -       -       discard
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual
lmtp      unix  -       -       y       -       -       lmtp
anvil     unix  -       -       y       -       1       anvil
scache    unix  -       -       y       -       1       scache
postlog   unix-dgram n  -       n       -       1       postlogd
#
# ====================================================================
# Interfaces to non-Postfix software. Be sure to examine the manual
# pages of the non-Postfix software to find out what options it wants.
#
# Many of the following services use the Postfix pipe(8) delivery
# agent.  See the pipe(8) man page for information about ${recipient}
# and other message envelope options.
# ====================================================================
#
# maildrop. See the Postfix MAILDROP_README file for details.
# Also specify in main.cf: maildrop_destination_recipient_limit=1
#
maildrop  unix  -       n       n       -       -       pipe
  flags=DRXhu user=vmail argv=/usr/bin/maildrop -d ${recipient}
#
# ====================================================================
#
# Recent Cyrus versions can use the existing "lmtp" master.cf entry.
#
# Specify in cyrus.conf:
#   lmtp    cmd="lmtpd -a" listen="localhost:lmtp" proto=tcp4
#
# Specify in main.cf one or more of the following:
#  mailbox_transport = lmtp:inet:localhost
#  virtual_transport = lmtp:inet:localhost
#
# ====================================================================
#
# Cyrus 2.1.5 (Amos Gouaux)
# Also specify in main.cf: cyrus_destination_recipient_limit=1
#
#cyrus     unix  -       n       n       -       -       pipe
#  flags=DRX user=cyrus argv=/cyrus/bin/deliver -e -r ${sender} -m ${extension} ${user}
#
# ====================================================================
# Old example of delivery via Cyrus.
#
#old-cyrus unix  -       n       n       -       -       pipe
#  flags=R user=cyrus argv=/cyrus/bin/deliver -e -m ${extension} ${user}
#
# ====================================================================
#
# See the Postfix UUCP_README file for configuration details.
#
uucp      unix  -       n       n       -       -       pipe
  flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
#
# Other external delivery methods.
#
ifmail    unix  -       n       n       -       -       pipe
  flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
bsmtp     unix  -       n       n       -       -       pipe
  flags=Fq. user=bsmtp argv=/usr/lib/bsmtp/bsmtp -t$nexthop -f$sender $recipient
scalemail-backend unix -       n       n       -       2       pipe
  flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store ${nexthop} ${user} ${extension}
mailman   unix  -       n       n       -       -       pipe
  flags=FRX user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py ${nexthop} ${user}
dovecot   unix  -       n       n       -       -       pipe
  flags=DRhu user=vmail:vmail argv=/usr/lib/dovecot/deliver -f ${sender} -d ${user}@${nexthop}

# added by diab
spfcheck  unix  -       n       n       -       0       spawn
                   user=policyd-spf argv=/usr/sbin/postfix-policyd-spf-perl


amavis unix - - - - 2 smtp
        -o smtp_data_done_timeout=1200
        -o smtp_send_xforward_command=yes
        -o smtp_bind_address=


127.0.0.1:10025 inet n - n - - smtpd
        -o content_filter=
        -o local_recipient_maps=
        -o relay_recipient_maps=
        -o smtpd_restriction_classes=
        -o smtpd_client_restrictions=
        -o smtpd_helo_restrictions=
        -o smtpd_sender_restrictions=
        -o smtpd_recipient_restrictions=permit_mynetworks,reject
        -o smtpd_end_of_data_restrictions=
        -o mynetworks=127.0.0.0/8
        -o strict_rfc821_envelopes=yes
        -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks
        -o smtp_send_xforward_command=yes
        -o disable_dns_lookups=yes
        -o address_verify_virtual_transport=$virtual_transport
        -o address_verify_transport_maps=$transport_maps


127.0.0.1:10027 inet n - n - - smtpd
        -o content_filter=
        -o local_recipient_maps=
        -o relay_recipient_maps=
        -o smtpd_restriction_classes=
        -o smtpd_client_restrictions=
        -o smtpd_helo_restrictions=
        -o smtpd_sender_restrictions=
        -o smtpd_recipient_restrictions=permit_mynetworks,reject
        -o smtpd_end_of_data_restrictions=
        -o mynetworks=127.0.0.0/8
        -o strict_rfc821_envelopes=yes
        -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks
        -o smtp_send_xforward_command=yes
        -o disable_dns_lookups=yes
        -o address_verify_virtual_transport=$virtual_transport
        -o address_verify_transport_maps=$transport_maps
        -o milter_default_action=accept
        -o milter_macro_daemon_name=ORIGINATING

amavis 50-user

Code:

use strict;

#
# Place your configuration directives here.  They will override those in
# earlier files.
#
# See /usr/share/doc/amavisd-new/ for documentation and examples of
# the directives you can use in this file
#

@bypass_virus_checks_maps = (
   \%bypass_virus_checks, \@bypass_virus_checks_acl, \$bypass_virus_checks_re);

@bypass_spam_checks_maps = (
   \%bypass_spam_checks, \@bypass_spam_checks_acl, \$bypass_spam_checks_re);

#  
# Database connection settings
#

@lookup_sql_dsn =
   ( ['DBI:mysql:database=dbispconfig;host=127.0.0.1;port=3306', 'xxxx', 'xxxx'] );

# @storage_sql_dsn = @lookup_sql_dsn;  # none, same, or separate database
#$sql_select_policy = 'SELECT "Y" as local FROM mail_domain WHERE CONCAT("@",domain) IN (%k)';
# $banned_files_quarantine_method = 'sql';
# $spam_quarantine_method         = 'sql';

#
# SQL Select statements
#

$sql_select_policy =
   'SELECT *,spamfilter_users.id'.
   ' FROM spamfilter_users LEFT JOIN spamfilter_policy ON spamfilter_users.policy_id=spamfilter_policy.id'.
   ' WHERE spamfilter_users.email IN (%k) AND spamfilter_users.policy_id != 0'.
   ' ORDER BY spamfilter_users.priority DESC';


$sql_select_white_black_list = 'SELECT wb FROM spamfilter_wblist'.
    ' WHERE (spamfilter_wblist.rid=?) AND (spamfilter_wblist.email IN (%k)) AND (spamfilter_wblist.active="y")' .
    ' ORDER BY spamfilter_wblist.priority DESC';

#
# Quarantine settings
#

$final_virus_destiny = D_BOUNCE;
$final_spam_destiny = D_DISCARD;
$final_banned_destiny = D_BOUNCE;
$final_bad_header_destiny = D_PASS;

# Default settings, we set this very high to not filter out emails accidentally
$sa_spam_subject_tag = '***SPAM*** ';
$sa_tag_level_deflt  = 20.0;  # add spam info headers if at, or above that level
$sa_tag2_level_deflt = 60.0; # add 'spam detected' headers at that level
$sa_kill_level_deflt = 60.0; # triggers spam evasive actions
$sa_dsn_cutoff_level = 100;   # spam level beyond which a DSN is not sent

#
# Disable spam and virus notifications for the admin user.
# Can be overridden by the policies in mysql
#

$virus_admin = undef;
$spam_admin = undef;


#
# Enable Logging
#

$DO_SYSLOG = 1;
$LOGFILE = "/var/log/amavis.log";  # (defaults to empty, no log)

# Set the log_level to 5 for debugging
$log_level = 0;                # (defaults to 0)

$inet_socket_port = [10024,10026];

# :* = send to incoming Port + 1
$forward_method = 'smtp:127.0.0.1:*';
$notify_method = 'smtp:127.0.0.1:*';
$interface_policy{'10026'} = 'ORIGINATING';
$policy_bank{'ORIGINATING'} = {
  originating => 1,
};

# IP-Addresses for internal networks => load policy MYNETS
# - requires -o smtp_send_xforward_command=yes in postfix master.cf
@mynetworks = qw(0.0.0.0/8 127.0.0.0/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 [::1] [FE80::]/10 [FEC0::]/10);

# Allow SMTP access from IPs in @inet_acl to amvisd SMTP Port
@inet_acl = qw( 127.0.0.1 [::1] 192.168.0.0/16 );

# DKIM
$enable_dkim_verification = 1;
$enable_dkim_signing = 1; # load DKIM signing code
$signed_header_fields{'received'} = 0;  # turn off signing of Received
@dkim_signature_options_bysender_maps = (
{ '.' => { ttl => 21*24*3600, c => 'relaxed/simple' } } );

#------------ Do not modify anything below this line -------------
1;  # insure a defined return

amavis dkim conf

Code:

dkim_key('xxxx.com', 'default', '/var/lib/amavis/dkim/xxxx.com.private');

 

Where does the php mail() function drop the emails?
If they're dropped in Postfix, are they sent to the outside world or delivered locally?
I don't think emails that stay local are being signed, only if they're relayed to another host.

 

The PHP mail() function uses sendmail to send mail IIRC, but the issue here is that mails that are send from the machine itself are not getting signed with DKIM as they don't seem to pass amavis for outbound

 

Ok, let me rephrase then.
Where does sendmail drop the emails generated with the php mail() function?
Into Postfix' queue or somewhere else?

It makes no sense that they don't get signed if they're dropped into Postfix when other outbound emails do get signed.
That makes me think they don't pass through Postfix and therefor Amavis at all.

 

The one that gets signed (i.e. by connecting to localhost 25) has this header:

Code:

Received: from localhost (localhost.localdomain [127.0.0.1])
by xx (Postfix) with SMTP id 30889194D283
    for <xx@xx>; Sat,  3 Dec 2022 13:20:02 +0100 (CET)

the one that is not signed is like this (sent by connecting to localhost:25):

Code:

    by localhost (xx [127.0.0.1]) (amavisd-new, port 10024)
    with LMTP id lNA_NRmLDnwq for <xx@xx>;
    Sat,  3 Dec 2022 13:18:36 +0100 (CET)
Received: by xx (Postfix, from userid 5013)
    id 625BB194D29A; Sat,  3 Dec 2022 13:18:36 +0100 (CET)

I ran an strace on the php process and it looks like mail() runs sendmail directly in the background

Code:

[pid 3133130] execve("/usr/sbin/sendmail", ["/usr/sbin/sendmail", "-t", "-i"], 0x55a9ecbfc6b0 /* 16 vars */) = 0

 

+1 this does not get signed:

Code:

echo -en 'From: webmaster@xx\nTo: xx@xx\nSubject: test\n\nhello\n.' | sendmail -t -i -f webmaster@xx

however, this does

Code:

# telnet 0 25
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.
220 xx ESMTP Postfix (Debian/GNU)
helo localhost
250 xx
mail from: webmaster@xx
250 2.1.0 Ok
rcpt to: xx@xx
250 2.1.5 Ok
data
354 End data with <CR><LF>.<CR><LF>
abc
.
250 2.0.0 Ok: queued as 8C61C194D1DE
quit
221 2.0.0 Bye
Connection closed by foreign host.

 

Trace the emails in the maillogs by id to see how they pass through Postfix (and Amavis) before going outside.
The headers you posted look somewhat incomplete to me and definitely don't show the whole path.

 

Is there a reason why you're using Amavis and not Rspamd?
Recently I had issues with Postfix and Amavis after implementing SRS where emails looped between Postfix and Amavis.
Not the same as you are facing but I solved it by replacing Amavis with Rspamd:
https://www.howtoforge.com/replacing-amavisd-with-rspamd-in-ispconfig/

 

Thanks again for the response. I use amavisd because I am familiar with it and the howto I was using to set up the server used that, too. Now that it's in production I'd prefer to fix it but I may replace it with rspamd if we can't figure it out.

Anyhow, here are the full headers:

Code:

Return-Path: <webmaster@xxxxx>
Delivered-To: xxxxx@xxxxx
Received: from xxxxx
    by xxxxx with LMTP
    id 4tYtAGdHi2PX4C8Ac2pvdg
    (envelope-from <webmaster@xxxxx>)
    for <xxxxx@xxxxx>; Sat, 03 Dec 2022 13:56:07 +0100
Received: from localhost (localhost.localdomain [127.0.0.1])
    by xxxxx (Postfix) with ESMTP id EE765194D29D
    for <xxxxx@xxxxx>; Sat,  3 Dec 2022 13:56:06 +0100 (CET)
X-Virus-Scanned: Debian amavisd-new at xxxxx
Received: from xxxxx ([127.0.0.1])
    by localhost (xxxxx [127.0.0.1]) (amavisd-new, port 10024)
    with LMTP id NClEPbvz1vCt for <xxxxx@xxxxx>;
    Sat,  3 Dec 2022 13:56:06 +0100 (CET)
Received: by xxxxx (Postfix, from userid 0)
    id BB9F2194D29E; Sat,  3 Dec 2022 13:56:06 +0100 (CET)
From: webmaster@xxxxx
To: xxxxx@xxxxx
Subject: test
Message-Id: <20221203125606.BB9F2194D29E@xxxxx>
Date: Sat,  3 Dec 2022 13:56:06 +0100 (CET)

mail server logs with ID: BB9F2194D29E

Code:

Dec  3 12:56:06 xxxxx postfix/pickup[3130767]: BB9F2194D29E: uid=0 from=<webmaster@xxxxx>
Dec  3 12:56:06 xxxxx postfix/cleanup[3137744]: BB9F2194D29E: message-id=<20221203125606.BB9F2194D29E@xxxxx>
Dec  3 12:56:06 xxxxx postfix/qmgr[2721401]: BB9F2194D29E: from=<webmaster@xxxxx>, size=307, nrcpt=1 (queue active)
Dec  3 12:56:06 xxxxx postfix/cleanup[3137744]: EE765194D29D: message-id=<20221203125606.BB9F2194D29E@xxxxx>
Dec  3 12:56:06 xxxxx amavis[3135506]: (3135506-01) Passed CLEAN {RelayedInbound}, [127.0.0.1] <webmaster@xxxxx> -> <xxxxx@xxxxx>, Message-ID: <20221203125606.BB9F2194D29E@xxxxx>, mail_id: NClEPbvz1vCt, Hits: -0.001, size: 305, queued_as: EE765194D29D, 201 ms
Dec  3 12:56:06 xxxxx postfix/lmtp[3137746]: BB9F2194D29E: to=<xxxxx@xxxxx>, relay=127.0.0.1[127.0.0.1]:10024, delay=0.23, delays=0.02/0.01/0/0.2, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as EE765194D29D)
Dec  3 12:56:06 xxxxx postfix/qmgr[2721401]: BB9F2194D29E: removed
Dec  3 12:56:07 xxxxx dovecot: lmtp(xxxxx@xxxxx)<3137751><4tYtAGdHi2PX4C8Ac2pvdg>: sieve: msgid=<20221203125606.BB9F2194D29E@xxxxx>: stored mail into mailbox 'INBOX'

On the other hand, the one that gets signed looks like this:

Code:

Dec  3 12:33:36 host3 amavis[2885068]: (2885068-20) Passed CLEAN {RelayedInternal}, ORIGINATING LOCAL [127.0.0.1] <xxxxx@xxxxx> -> <xxxxx@xxxxx>, Message-ID: <20221203123332.8C61C194D1DE@xxxxx>, mail_id: PMLC8Pahuq5M, Hits: 1.974, size: 334, queued_as: 1FB4D194D29D, dkim_new=default:xxxxx, 163 ms

Maybe the reason is that Amavisd treats direct mail as "relayedinbound" as opposed to "relayedinternal" and it should be somehow configured to enable dkim signing there, too.

 

yaay, I figured it out. Enabling debug logs in amavis showed:

Code:

Dec  3 13:42:08 xxxxx amavis[3148219]: (3148219-01) dkim: not signing mail which is not originating from our site

So I had to enable "originating" for the "MYNETS" policy bank in amavis 50-user.conf and it started working

Code:

$policy_bank{'MYNETS'} = {
  originating => 1,
};

I've verified and it's not signing random e-mails from outside so it looks like it works as intended, local mail / trusted hosts only.

 

Glad you found the solution.