DKIM for locally generated mail isn't working (Debian11, Ispconfig 3.2, Amavisd-new)

Discussion in 'General' started by diab, Dec 2, 2022.

  1. diab

    diab New Member

    I'm operating a Debian11 server with Ispconfig 3.2. DKIM signing is done by amavisd-new-2.11.1 (20181009).
    I've noticed that locally generated outgoing mail (directly with the php mail() function from the domain's associated website) is not signed by DKIM. It works when it's sent through SMTP (connecting to localhost:25, unauthenticated) and verifies ok so DKIM is set up properly, keys work, etc.
    The return-path and from headers are OK in both cases (i.e. mail is coming from webmaster@domain as they should).
    What am I missing to fix this? I recall that there was an earlier one liner patch to amavisd-new to fix some sort of local DKIM signing issue but I'm not sure if it's relevant.
     
  2. pyte

    pyte Well-Known Member HowtoForge Supporter

    There are multiple ways to achieve this. The Issue is that locally generated mails are not getting handed over to amavis.
    One way could be to change the master.cf to something like this:
    Code:
    pickup  fifo    n       -       n       60      1       pickup
      -o content_filter=  // COMMENT THIS OUT
      -o receive_override_options=
    When you comment the line out it should work
     
  3. diab

    diab New Member

    Thank you for the quick response! Unfortunately that's not the case, the pickup line (ha ha) has no extra options and amavisd does scan it, i.e. the non dkim-signed local message has an "X-Virus-Scanned: Debian amavisd-new at hostname" header but no DKIM.
     
  4. pyte

    pyte Well-Known Member HowtoForge Supporter

    Oh well that is a different issue than. Can you post your master.cf and amavis config(i assume it is called 50_user.conf or something similar)?
     
  5. diab

    diab New Member

    Here they are - it's mostly stock except for a small change in master.cf to add policy-spf

    master.cf
    Code:
    
    #
    # Postfix master process configuration file.  For details on the format
    # of the file, see the master(5) manual page (command: "man 5 master" or
    # on-line: http://www.postfix.org/master.5.html).
    #
    # Do not forget to execute "postfix reload" after editing this file.
    #
    # ==========================================================================
    # service type  private unpriv  chroot  wakeup  maxproc command + args
    #               (yes)   (yes)   (no)    (never) (100)
    # ==========================================================================
    smtp      inet  n       -       y       -       -       smtpd
    #smtp      inet  n       -       y       -       1       postscreen
    #smtpd     pass  -       -       y       -       -       smtpd
    #dnsblog   unix  -       -       y       -       0       dnsblog
    #tlsproxy  unix  -       -       y       -       0       tlsproxy
    submission inet n       -       y       -       -       smtpd
      -o syslog_name=postfix/submission
      -o smtpd_tls_security_level=encrypt
      -o smtpd_sasl_auth_enable=yes
    #  -o smtpd_tls_auth_only=yes
    #  -o smtpd_reject_unlisted_recipient=no
    #  -o smtpd_client_restrictions=$mua_client_restrictions
    #  -o smtpd_helo_restrictions=$mua_helo_restrictions
    #  -o smtpd_sender_restrictions=$mua_sender_restrictions
    #  -o smtpd_recipient_restrictions=
      -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
    #  -o milter_macro_daemon_name=ORIGINATING
    smtps     inet  n       -       y       -       -       smtpd
      -o syslog_name=postfix/smtps
      -o smtpd_tls_wrappermode=yes
      -o smtpd_sasl_auth_enable=yes
    #  -o smtpd_reject_unlisted_recipient=no
    #  -o smtpd_client_restrictions=$mua_client_restrictions
    #  -o smtpd_helo_restrictions=$mua_helo_restrictions
    #  -o smtpd_sender_restrictions=$mua_sender_restrictions
    #  -o smtpd_recipient_restrictions=
      -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
    #  -o milter_macro_daemon_name=ORIGINATING
    #628       inet  n       -       y       -       -       qmqpd
    pickup    unix  n       -       y       60      1       pickup
    cleanup   unix  n       -       y       -       0       cleanup
    qmgr      unix  n       -       n       300     1       qmgr
    #qmgr     unix  n       -       n       300     1       oqmgr
    tlsmgr    unix  -       -       y       1000?   1       tlsmgr
    rewrite   unix  -       -       y       -       -       trivial-rewrite
    bounce    unix  -       -       y       -       0       bounce
    defer     unix  -       -       y       -       0       bounce
    trace     unix  -       -       y       -       0       bounce
    verify    unix  -       -       y       -       1       verify
    flush     unix  n       -       y       1000?   0       flush
    proxymap  unix  -       -       n       -       -       proxymap
    proxywrite unix -       -       n       -       1       proxymap
    smtp      unix  -       -       y       -       -       smtp
    relay     unix  -       -       y       -       -       smtp
            -o syslog_name=postfix/$service_name
    #       -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
    showq     unix  n       -       y       -       -       showq
    error     unix  -       -       y       -       -       error
    retry     unix  -       -       y       -       -       error
    discard   unix  -       -       y       -       -       discard
    local     unix  -       n       n       -       -       local
    virtual   unix  -       n       n       -       -       virtual
    lmtp      unix  -       -       y       -       -       lmtp
    anvil     unix  -       -       y       -       1       anvil
    scache    unix  -       -       y       -       1       scache
    postlog   unix-dgram n  -       n       -       1       postlogd
    #
    # ====================================================================
    # Interfaces to non-Postfix software. Be sure to examine the manual
    # pages of the non-Postfix software to find out what options it wants.
    #
    # Many of the following services use the Postfix pipe(8) delivery
    # agent.  See the pipe(8) man page for information about ${recipient}
    # and other message envelope options.
    # ====================================================================
    #
    # maildrop. See the Postfix MAILDROP_README file for details.
    # Also specify in main.cf: maildrop_destination_recipient_limit=1
    #
    maildrop  unix  -       n       n       -       -       pipe
      flags=DRXhu user=vmail argv=/usr/bin/maildrop -d ${recipient}
    #
    # ====================================================================
    #
    # Recent Cyrus versions can use the existing "lmtp" master.cf entry.
    #
    # Specify in cyrus.conf:
    #   lmtp    cmd="lmtpd -a" listen="localhost:lmtp" proto=tcp4
    #
    # Specify in main.cf one or more of the following:
    #  mailbox_transport = lmtp:inet:localhost
    #  virtual_transport = lmtp:inet:localhost
    #
    # ====================================================================
    #
    # Cyrus 2.1.5 (Amos Gouaux)
    # Also specify in main.cf: cyrus_destination_recipient_limit=1
    #
    #cyrus     unix  -       n       n       -       -       pipe
    #  flags=DRX user=cyrus argv=/cyrus/bin/deliver -e -r ${sender} -m ${extension} ${user}
    #
    # ====================================================================
    # Old example of delivery via Cyrus.
    #
    #old-cyrus unix  -       n       n       -       -       pipe
    #  flags=R user=cyrus argv=/cyrus/bin/deliver -e -m ${extension} ${user}
    #
    # ====================================================================
    #
    # See the Postfix UUCP_README file for configuration details.
    #
    uucp      unix  -       n       n       -       -       pipe
      flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
    #
    # Other external delivery methods.
    #
    ifmail    unix  -       n       n       -       -       pipe
      flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
    bsmtp     unix  -       n       n       -       -       pipe
      flags=Fq. user=bsmtp argv=/usr/lib/bsmtp/bsmtp -t$nexthop -f$sender $recipient
    scalemail-backend unix -       n       n       -       2       pipe
      flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store ${nexthop} ${user} ${extension}
    mailman   unix  -       n       n       -       -       pipe
      flags=FRX user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py ${nexthop} ${user}
    dovecot   unix  -       n       n       -       -       pipe
      flags=DRhu user=vmail:vmail argv=/usr/lib/dovecot/deliver -f ${sender} -d ${user}@${nexthop}
    
    # added by diab
    spfcheck  unix  -       n       n       -       0       spawn
                       user=policyd-spf argv=/usr/sbin/postfix-policyd-spf-perl
    
    
    amavis unix - - - - 2 smtp
            -o smtp_data_done_timeout=1200
            -o smtp_send_xforward_command=yes
            -o smtp_bind_address=
    
    
    127.0.0.1:10025 inet n - n - - smtpd
            -o content_filter=
            -o local_recipient_maps=
            -o relay_recipient_maps=
            -o smtpd_restriction_classes=
            -o smtpd_client_restrictions=
            -o smtpd_helo_restrictions=
            -o smtpd_sender_restrictions=
            -o smtpd_recipient_restrictions=permit_mynetworks,reject
            -o smtpd_end_of_data_restrictions=
            -o mynetworks=127.0.0.0/8
            -o strict_rfc821_envelopes=yes
            -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks
            -o smtp_send_xforward_command=yes
            -o disable_dns_lookups=yes
            -o address_verify_virtual_transport=$virtual_transport
            -o address_verify_transport_maps=$transport_maps
    
    
    127.0.0.1:10027 inet n - n - - smtpd
            -o content_filter=
            -o local_recipient_maps=
            -o relay_recipient_maps=
            -o smtpd_restriction_classes=
            -o smtpd_client_restrictions=
            -o smtpd_helo_restrictions=
            -o smtpd_sender_restrictions=
            -o smtpd_recipient_restrictions=permit_mynetworks,reject
            -o smtpd_end_of_data_restrictions=
            -o mynetworks=127.0.0.0/8
            -o strict_rfc821_envelopes=yes
            -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks
            -o smtp_send_xforward_command=yes
            -o disable_dns_lookups=yes
            -o address_verify_virtual_transport=$virtual_transport
            -o address_verify_transport_maps=$transport_maps
            -o milter_default_action=accept
            -o milter_macro_daemon_name=ORIGINATING
    
    amavis 50-user
    Code:
    use strict;
    
    #
    # Place your configuration directives here.  They will override those in
    # earlier files.
    #
    # See /usr/share/doc/amavisd-new/ for documentation and examples of
    # the directives you can use in this file
    #
    
    @bypass_virus_checks_maps = (
       \%bypass_virus_checks, \@bypass_virus_checks_acl, \$bypass_virus_checks_re);
    
    @bypass_spam_checks_maps = (
       \%bypass_spam_checks, \@bypass_spam_checks_acl, \$bypass_spam_checks_re);
    
    #  
    # Database connection settings
    #
    
    @lookup_sql_dsn =
       ( ['DBI:mysql:database=dbispconfig;host=127.0.0.1;port=3306', 'xxxx', 'xxxx'] );
    
    # @storage_sql_dsn = @lookup_sql_dsn;  # none, same, or separate database
    #$sql_select_policy = 'SELECT "Y" as local FROM mail_domain WHERE CONCAT("@",domain) IN (%k)';
    # $banned_files_quarantine_method = 'sql';
    # $spam_quarantine_method         = 'sql';
    
    #
    # SQL Select statements
    #
    
    $sql_select_policy =
       'SELECT *,spamfilter_users.id'.
       ' FROM spamfilter_users LEFT JOIN spamfilter_policy ON spamfilter_users.policy_id=spamfilter_policy.id'.
       ' WHERE spamfilter_users.email IN (%k) AND spamfilter_users.policy_id != 0'.
       ' ORDER BY spamfilter_users.priority DESC';
    
    
    $sql_select_white_black_list = 'SELECT wb FROM spamfilter_wblist'.
        ' WHERE (spamfilter_wblist.rid=?) AND (spamfilter_wblist.email IN (%k)) AND (spamfilter_wblist.active="y")' .
        ' ORDER BY spamfilter_wblist.priority DESC';
    
    #
    # Quarantine settings
    #
    
    $final_virus_destiny = D_BOUNCE;
    $final_spam_destiny = D_DISCARD;
    $final_banned_destiny = D_BOUNCE;
    $final_bad_header_destiny = D_PASS;
    
    # Default settings, we set this very high to not filter out emails accidentally
    $sa_spam_subject_tag = '***SPAM*** ';
    $sa_tag_level_deflt  = 20.0;  # add spam info headers if at, or above that level
    $sa_tag2_level_deflt = 60.0; # add 'spam detected' headers at that level
    $sa_kill_level_deflt = 60.0; # triggers spam evasive actions
    $sa_dsn_cutoff_level = 100;   # spam level beyond which a DSN is not sent
    
    #
    # Disable spam and virus notifications for the admin user.
    # Can be overridden by the policies in mysql
    #
    
    $virus_admin = undef;
    $spam_admin = undef;
    
    
    #
    # Enable Logging
    #
    
    $DO_SYSLOG = 1;
    $LOGFILE = "/var/log/amavis.log";  # (defaults to empty, no log)
    
    # Set the log_level to 5 for debugging
    $log_level = 0;                # (defaults to 0)
    
    $inet_socket_port = [10024,10026];
    
    # :* = send to incoming Port + 1
    $forward_method = 'smtp:127.0.0.1:*';
    $notify_method = 'smtp:127.0.0.1:*';
    $interface_policy{'10026'} = 'ORIGINATING';
    $policy_bank{'ORIGINATING'} = {
      originating => 1,
    };
    
    # IP-Addresses for internal networks => load policy MYNETS
    # - requires -o smtp_send_xforward_command=yes in postfix master.cf
    @mynetworks = qw(0.0.0.0/8 127.0.0.0/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 [::1] [FE80::]/10 [FEC0::]/10);
    
    # Allow SMTP access from IPs in @inet_acl to amvisd SMTP Port
    @inet_acl = qw( 127.0.0.1 [::1] 192.168.0.0/16 );
    
    # DKIM
    $enable_dkim_verification = 1;
    $enable_dkim_signing = 1; # load DKIM signing code
    $signed_header_fields{'received'} = 0;  # turn off signing of Received
    @dkim_signature_options_bysender_maps = (
    { '.' => { ttl => 21*24*3600, c => 'relaxed/simple' } } );
    
    #------------ Do not modify anything below this line -------------
    1;  # insure a defined return
    
    amavis dkim conf
    Code:
    dkim_key('xxxx.com', 'default', '/var/lib/amavis/dkim/xxxx.com.private');
    
     
  6. remkoh

    remkoh Active Member

    Where does the php mail() function drop the emails?
    If they're dropped in Postfix, are they sent to the outside world or delivered locally?
    I don't think emails that stay local are being signed, only if they're relayed to another host.
     
  7. pyte

    pyte Well-Known Member HowtoForge Supporter

    The PHP mail() function uses sendmail to send mail IIRC, but the issue here is that mails that are send from the machine itself are not getting signed with DKIM as they don't seem to pass amavis for outbound
     
  8. remkoh

    remkoh Active Member

    Ok, let me rephrase then.
    Where does sendmail drop the emails generated with the php mail() function?
    Into Postfix' queue or somewhere else?

    It makes no sense that they don't get signed if they're dropped into Postfix when other outbound emails do get signed.
    That makes me think they don't pass through Postfix and therefor Amavis at all.
     
    Last edited: Dec 3, 2022
  9. diab

    diab New Member

    The one that gets signed (i.e. by connecting to localhost 25) has this header:
    Code:
    Received: from localhost (localhost.localdomain [127.0.0.1])
    by xx (Postfix) with SMTP id 30889194D283
        for <xx@xx>; Sat,  3 Dec 2022 13:20:02 +0100 (CET)
    
    the one that is not signed is like this (sent by connecting to localhost:25):
    Code:
        by localhost (xx [127.0.0.1]) (amavisd-new, port 10024)
        with LMTP id lNA_NRmLDnwq for <xx@xx>;
        Sat,  3 Dec 2022 13:18:36 +0100 (CET)
    Received: by xx (Postfix, from userid 5013)
        id 625BB194D29A; Sat,  3 Dec 2022 13:18:36 +0100 (CET)
    
    I ran an strace on the php process and it looks like mail() runs sendmail directly in the background
    Code:
    [pid 3133130] execve("/usr/sbin/sendmail", ["/usr/sbin/sendmail", "-t", "-i"], 0x55a9ecbfc6b0 /* 16 vars */) = 0
    
     
  10. diab

    diab New Member

    +1 this does not get signed:
    Code:
    echo -en 'From: webmaster@xx\nTo: xx@xx\nSubject: test\n\nhello\n.' | sendmail -t -i -f webmaster@xx
    
    however, this does
    Code:
    # telnet 0 25
    Trying 0.0.0.0...
    Connected to 0.
    Escape character is '^]'.
    220 xx ESMTP Postfix (Debian/GNU)
    helo localhost
    250 xx
    mail from: webmaster@xx
    250 2.1.0 Ok
    rcpt to: xx@xx
    250 2.1.5 Ok
    data
    354 End data with <CR><LF>.<CR><LF>
    abc
    .
    250 2.0.0 Ok: queued as 8C61C194D1DE
    quit
    221 2.0.0 Bye
    Connection closed by foreign host.
    
     
  11. remkoh

    remkoh Active Member

    Trace the emails in the maillogs by id to see how they pass through Postfix (and Amavis) before going outside.
    The headers you posted look somewhat incomplete to me and definitely don't show the whole path.
     
  12. remkoh

    remkoh Active Member

  13. diab

    diab New Member

    Thanks again for the response. I use amavisd because I am familiar with it and the howto I was using to set up the server used that, too. Now that it's in production I'd prefer to fix it but I may replace it with rspamd if we can't figure it out.

    Anyhow, here are the full headers:

    Code:
    Return-Path: <webmaster@xxxxx>
    Delivered-To: xxxxx@xxxxx
    Received: from xxxxx
        by xxxxx with LMTP
        id 4tYtAGdHi2PX4C8Ac2pvdg
        (envelope-from <webmaster@xxxxx>)
        for <xxxxx@xxxxx>; Sat, 03 Dec 2022 13:56:07 +0100
    Received: from localhost (localhost.localdomain [127.0.0.1])
        by xxxxx (Postfix) with ESMTP id EE765194D29D
        for <xxxxx@xxxxx>; Sat,  3 Dec 2022 13:56:06 +0100 (CET)
    X-Virus-Scanned: Debian amavisd-new at xxxxx
    Received: from xxxxx ([127.0.0.1])
        by localhost (xxxxx [127.0.0.1]) (amavisd-new, port 10024)
        with LMTP id NClEPbvz1vCt for <xxxxx@xxxxx>;
        Sat,  3 Dec 2022 13:56:06 +0100 (CET)
    Received: by xxxxx (Postfix, from userid 0)
        id BB9F2194D29E; Sat,  3 Dec 2022 13:56:06 +0100 (CET)
    From: webmaster@xxxxx
    To: xxxxx@xxxxx
    Subject: test
    Message-Id: <20221203125606.BB9F2194D29E@xxxxx>
    Date: Sat,  3 Dec 2022 13:56:06 +0100 (CET)
    
    mail server logs with ID: BB9F2194D29E

    Code:
    Dec  3 12:56:06 xxxxx postfix/pickup[3130767]: BB9F2194D29E: uid=0 from=<webmaster@xxxxx>
    Dec  3 12:56:06 xxxxx postfix/cleanup[3137744]: BB9F2194D29E: message-id=<20221203125606.BB9F2194D29E@xxxxx>
    Dec  3 12:56:06 xxxxx postfix/qmgr[2721401]: BB9F2194D29E: from=<webmaster@xxxxx>, size=307, nrcpt=1 (queue active)
    Dec  3 12:56:06 xxxxx postfix/cleanup[3137744]: EE765194D29D: message-id=<20221203125606.BB9F2194D29E@xxxxx>
    Dec  3 12:56:06 xxxxx amavis[3135506]: (3135506-01) Passed CLEAN {RelayedInbound}, [127.0.0.1] <webmaster@xxxxx> -> <xxxxx@xxxxx>, Message-ID: <20221203125606.BB9F2194D29E@xxxxx>, mail_id: NClEPbvz1vCt, Hits: -0.001, size: 305, queued_as: EE765194D29D, 201 ms
    Dec  3 12:56:06 xxxxx postfix/lmtp[3137746]: BB9F2194D29E: to=<xxxxx@xxxxx>, relay=127.0.0.1[127.0.0.1]:10024, delay=0.23, delays=0.02/0.01/0/0.2, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as EE765194D29D)
    Dec  3 12:56:06 xxxxx postfix/qmgr[2721401]: BB9F2194D29E: removed
    Dec  3 12:56:07 xxxxx dovecot: lmtp(xxxxx@xxxxx)<3137751><4tYtAGdHi2PX4C8Ac2pvdg>: sieve: msgid=<20221203125606.BB9F2194D29E@xxxxx>: stored mail into mailbox 'INBOX'
    
    On the other hand, the one that gets signed looks like this:
    Code:
    Dec  3 12:33:36 host3 amavis[2885068]: (2885068-20) Passed CLEAN {RelayedInternal}, ORIGINATING LOCAL [127.0.0.1] <xxxxx@xxxxx> -> <xxxxx@xxxxx>, Message-ID: <20221203123332.8C61C194D1DE@xxxxx>, mail_id: PMLC8Pahuq5M, Hits: 1.974, size: 334, queued_as: 1FB4D194D29D, dkim_new=default:xxxxx, 163 ms
    
    Maybe the reason is that Amavisd treats direct mail as "relayedinbound" as opposed to "relayedinternal" and it should be somehow configured to enable dkim signing there, too.
     
  14. diab

    diab New Member

    yaay, I figured it out. Enabling debug logs in amavis showed:
    Code:
    Dec  3 13:42:08 xxxxx amavis[3148219]: (3148219-01) dkim: not signing mail which is not originating from our site
    
    So I had to enable "originating" for the "MYNETS" policy bank in amavis 50-user.conf and it started working
    Code:
    $policy_bank{'MYNETS'} = {
      originating => 1,
    };
    
    I've verified and it's not signing random e-mails from outside so it looks like it works as intended, local mail / trusted hosts only.
     
    Th0m and till like this.
  15. remkoh

    remkoh Active Member

    Glad you found the solution.
     
    diab likes this.

Share This Page