Hi. One particular user complains about missing emails. I had a look into it and found a specific line in the logs: ... postfix/cleanup[14324]: D4F5433CD61: discard: header ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; ... If the sender retries, it mostly does not appear in that second delivery attempt; instead, it flows through amavis, dovecot and gets delivered. Most email is delivered without issue, but this happens a view times on a weekly basis - roughly with about 5% of the specific sender's emails gets lost first time. It does not feel trustworthy, that's the biggest issue. Anyone that knows how i can relax postfix or make sure this sender's domain is treated as trusted source? (it's an office365 customer with own domain, so this probably comes up more often than I know of at the moment) Thanks, Martijn
Hi Jesse. Appreciate your response very much. I need to anonimize the log example, I'll post it later. Here the main.cf: Code: # See /usr/share/postfix/main.cf.dist for a commented, more complete version # Debian specific: Specifying a file name will cause the first # line of that file to be used as the name. The Debian default # is /etc/mailname. #myorigin = /etc/mailname smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu) biff = no # appending .domain is the MUA's job. append_dot_mydomain = no # Uncomment the next line to generate "delayed mail" warnings #delay_warning_time = 4h readme_directory = /usr/share/doc/postfix # See http://www.postfix.org/COMPATIBILITY_README.html -- default to 2 on # fresh installs. compatibility_level = 2 # TLS parameters smtpd_tls_cert_file = /etc/postfix/smtpd.cert smtpd_tls_key_file = /etc/postfix/smtpd.key smtpd_use_tls = yes smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache # See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for # information on enabling SSL in the smtp client. smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination myhostname = not.therealname.com alias_maps = hash:/etc/aliases, hash:/var/lib/mailman/data/aliases alias_database = hash:/etc/aliases, hash:/var/lib/mailman/data/aliases myorigin = /etc/mailname mydestination = not.therealname.com, localhost, localhost.localdomain relayhost = #mynetworks = 127.0.0.0/8 [::1]/128 mynetworks = 127.0.0.0/8 mailbox_size_limit = 0 recipient_delimiter = + inet_interfaces = all inet_protocols = ipv4 html_directory = /usr/share/doc/postfix/html virtual_alias_domains = virtual_alias_maps = hash:/var/lib/mailman/data/virtual-mailman, proxy:mysql:/etc/postfix/mysql-virtual_forwardings.cf, proxy:mysql:/etc/postfix/mysql-virtual_email2email.cf virtual_mailbox_domains = proxy:mysql:/etc/postfix/mysql-virtual_domains.cf virtual_mailbox_maps = proxy:mysql:/etc/postfix/mysql-virtual_mailboxes.cf virtual_mailbox_base = /var/vmail virtual_uid_maps = mysql:/etc/postfix/mysql-virtual_uids.cf virtual_gid_maps = mysql:/etc/postfix/mysql-virtual_gids.cf sender_bcc_maps = proxy:mysql:/etc/postfix/mysql-virtual_outgoing_bcc.cf smtpd_sasl_auth_enable = yes broken_sasl_auth_clients = yes smtpd_sasl_authenticated_header = yes smtpd_restriction_classes = greylisting greylisting = check_policy_service inet:127.0.0.1:10023 smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination, reject_invalid_hostname, reject_non_fqdn_hostname, reject_unknown_recipient_domain, reject_non_fqdn_recipient, reject_non_fqdn_sender, reject_unknown_sender_domain, reject_unknown_recipient_domain, reject_rbl_client zen.spamhaus.org, reject_rbl_client bl.spamcop.net, check_recipient_access mysql:/etc/postfix/mysql-virtual_recipient.cf, check_recipient_access mysql:/etc/postfix/mysql-virtual_policy_greylist.cf smtpd_tls_security_level = may transport_maps = hash:/var/lib/mailman/data/transport-mailman, proxy:mysql:/etc/postfix/mysql-virtual_transports.cf relay_domains = mysql:/etc/postfix/mysql-virtual_relaydomains.cf relay_recipient_maps = mysql:/etc/postfix/mysql-virtual_relayrecipientmaps.cf smtpd_sender_login_maps = proxy:mysql:/etc/postfix/mysql-virtual_sender_login_maps.cf proxy_read_maps = $local_recipient_maps $mydestination $virtual_alias_maps $virtual_alias_domains $sender_bcc_maps $virtual_mailbox_maps $virtual_mailbox_domains $relay_recipient_maps $relay_domains $canonical_maps $sender_canonical_maps $recipient_canonical_maps $relocated_maps $transport_maps $mynetworks $smtpd_sender_login_maps smtpd_helo_required = yes smtpd_helo_restrictions = permit_sasl_authenticated, permit_mynetworks, check_helo_access regexp:/etc/postfix/helo_access, reject_invalid_hostname, reject_non_fqdn_hostname, reject_invalid_helo_hostname, reject_unknown_helo_hostname, check_helo_access regexp:/etc/postfix/blacklist_helo smtpd_sender_restrictions = check_sender_access mysql:/etc/postfix/mysql-virtual_sender.cf regexp:/etc/postfix/tag_as_originating.re, permit_mynetworks, permit_sasl_authenticated, check_sender_access regexp:/etc/postfix/tag_as_foreign.re smtpd_client_restrictions = check_client_access mysql:/etc/postfix/mysql-virtual_client.cf smtpd_client_message_rate_limit = 100 maildrop_destination_concurrency_limit = 1 maildrop_destination_recipient_limit = 1 virtual_transport = dovecot header_checks = regexp:/etc/postfix/header_checks mime_header_checks = regexp:/etc/postfix/mime_header_checks nested_header_checks = regexp:/etc/postfix/nested_header_checks body_checks = regexp:/etc/postfix/body_checks owner_request_special = no smtp_tls_security_level = may smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3 smtpd_tls_protocols = !SSLv2,!SSLv3 smtp_tls_protocols = !SSLv2,!SSLv3 smtpd_tls_exclude_ciphers = RC4, aNULL smtp_tls_exclude_ciphers = RC4, aNULL dovecot_destination_recipient_limit = 1 smtpd_sasl_type = dovecot smtpd_sasl_path = private/auth content_filter = amavis:[127.0.0.1]:10024 receive_override_options = no_address_mappings message_size_limit = 0
Anonimized example first try: Code: May 3 13:44:54 anonhost postfix/smtpd[12481]: connect from mail-eopbgr20040.outbound.protection.outlook.com[40.107.2.40] May 3 13:44:54 anonhost postfix/smtpd[12481]: NOQUEUE: filter: RCPT from mail-eopbgr20040.outbound.protection.outlook.com[40.107.2.40]: <[email protected]>: Sender address triggers FILTER amavis:[127.0.0.1]:10026; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<EUR02-VE1-obe.outbound.protection.outlook.com> May 3 13:44:54 anonhost postfix/smtpd[12481]: NOQUEUE: filter: RCPT from mail-eopbgr20040.outbound.protection.outlook.com[40.107.2.40]: <[email protected]>: Sender address triggers FILTER amavis:[127.0.0.1]:10024; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<EUR02-VE1-obe.outbound.protection.outlook.com> May 3 13:44:54 anonhost postfix/smtpd[12481]: D4F5433CD61: client=mail-eopbgr20040.outbound.protection.outlook.com[40.107.2.40] May 3 13:44:54 anonhost postfix/cleanup[14324]: D4F5433CD61: discard: header ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;? b=Nk6q2Krwjvc+6N9knViZmITQeSuR4KgXb9N2qQlNnIHxA0ePg5gvTt4o9v0PAIYTaJ6IlmBej34UZAYn287rF2ubrqRzyJ211p+v9OpLuyv7017gYC5PZIO9uei from mail-eopbgr20040.outbound.protection.outlook.com[40.107.2.40]; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<EUR02-VE1-obe.outbound.protection.outlook.com> May 3 13:44:55 anonhost postfix/smtpd[12481]: disconnect from mail-eopbgr20040.outbound.protection.outlook.com[40.107.2.40] ehlo=2 starttls=1 mail=1 rcpt=1 data=1 quit=1 commands=7
Anonimized example second try + delivery to 2nd recipient (sender tries to make sure it gets delivered...). I could not send the full log, beacsue of all debug statements - too large for the forum: Code: May 3 14:33:33 anonhost postfix/smtpd[17674]: connect from mail-eopbgr30071.outbound.protection.outlook.com[40.107.3.71] May 3 14:33:33 anonhost postfix/smtpd[17674]: NOQUEUE: filter: RCPT from mail-eopbgr30071.outbound.protection.outlook.com[40.107.3.71]: <[email protected]>: Sender address triggers FILTER amavis:[127.0.0.1]:10026; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<EUR03-AM5-obe.outbound.protection.outlook.com> May 3 14:33:33 anonhost postfix/smtpd[17674]: NOQUEUE: filter: RCPT from mail-eopbgr30071.outbound.protection.outlook.com[40.107.3.71]: <[email protected]>: Sender address triggers FILTER amavis:[127.0.0.1]:10024; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<EUR03-AM5-obe.outbound.protection.outlook.com> May 3 14:33:33 anonhost postfix/smtpd[17674]: C04D8604320: client=mail-eopbgr30071.outbound.protection.outlook.com[40.107.3.71] May 3 14:33:33 anonhost postfix/smtpd[17674]: C04D8604320: filter: RCPT from mail-eopbgr30071.outbound.protection.outlook.com[40.107.3.71]: <[email protected]>: Sender address triggers FILTER amavis:[127.0.0.1]:10026; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<EUR03-AM5-obe.outbound.protection.outlook.com> May 3 14:33:33 anonhost postfix/smtpd[17674]: C04D8604320: filter: RCPT from mail-eopbgr30071.outbound.protection.outlook.com[40.107.3.71]: <[email protected]>: Sender address triggers FILTER amavis:[127.0.0.1]:10024; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<EUR03-AM5-obe.outbound.protection.outlook.com> May 3 14:33:33 anonhost postfix/cleanup[17647]: C04D8604320: message-id=<AM0PR06MB65612AB3A4339109547624F0F55B9@AM0PR06MB6561.eurprd06.prod.outlook.com> May 3 14:33:33 anonhost postfix/qmgr[818]: C04D8604320: from=<[email protected]>, size=132559, nrcpt=2 (queue active) May 3 14:33:33 anonhost amavis[17716]: Net::Server: 2021/05/03-14:33:33 CONNECT TCP Peer: "[127.0.0.1]:40720" Local: "[127.0.0.1]:10024" May 3 14:33:33 anonhost amavis[17716]: loaded base policy bank May 3 14:33:33 anonhost amavis[17716]: lookup_ip_acl (inet_acl) arr.obj: key="127.0.0.1" matches "127.0.0.1", result=1 May 3 14:33:33 anonhost amavis[17716]: process_request: fileno sock=15, STDIN=0, STDOUT=1 May 3 14:33:33 anonhost amavis[17716]: get_deadline switch_to_my_time(new request) - deadline in 480.0 s, set to 288.000 s May 3 14:33:33 anonhost amavis[17716]: prolong_timer switch_to_my_time(new request): timer 288, was 0, deadline in 480.0 s May 3 14:33:33 anonhost amavis[17716]: process_request: suggested_protocol="" on a TCP socket May 3 14:33:33 anonhost postfix/smtpd[17674]: disconnect from mail-eopbgr30071.outbound.protection.outlook.com[40.107.3.71] ehlo=2 starttls=1 mail=1 rcpt=2 data=1 quit=1 commands=8 May 3 14:33:36 anonhost postfix/smtpd[17771]: connect from localhost[127.0.0.1] May 3 14:33:36 anonhost postfix/smtpd[17771]: 77E47604481: client=localhost[127.0.0.1] May 3 14:33:36 anonhost postfix/cleanup[17384]: 77E47604481: message-id=<AM0PR06MB65612AB3A4339109547624F0F55B9@AM0PR06MB6561.eurprd06.prod.outlook.com> May 3 14:33:36 anonhost postfix/qmgr[818]: 77E47604481: from=<[email protected]>, size=133058, nrcpt=2 (queue active) May 3 14:33:36 anonhost postfix/smtp[17710]: C04D8604320: to=<[email protected]>, relay=127.0.0.1[127.0.0.1]:10024, delay=2.8, delays=0.14/0/0.01/2.7, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 77E47604481) May 3 14:33:36 anonhost postfix/smtp[17710]: C04D8604320: to=<[email protected]>, relay=127.0.0.1[127.0.0.1]:10024, delay=2.8, delays=0.14/0/0.01/2.7, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 77E47604481) May 3 14:33:36 anonhost postfix/qmgr[818]: C04D8604320: removed May 3 14:33:36 anonhost dovecot: lda([email protected]): sieve: msgid=? <AM0PR06MB65612AB3A4339109547624F0F55B9@AM0PR06MB6561.eurprd06.prod.outlook.com>: stored mail into mailbox 'INBOX' May 3 14:33:36 anonhost postfix/pipe[17775]: 77E47604481: to=<[email protected]>, relay=dovecot, delay=0.07, delays=0.02/0.01/0/0.04, dsn=2.0.0, status=sent (delivered via dovecot service (doveconf: Warning: /etc/dovecot/dovecot.conf line 71: Global setting mail_plugins won't change the s)) May 3 14:33:36 anonhost dovecot: lda([email protected]): sieve: msgid=? <AM0PR06MB65612AB3A4339109547624F0F55B9@AM0PR06MB6561.eurprd06.prod.outlook.com>: stored mail into mailbox 'INBOX' May 3 14:33:36 anonhost postfix/pipe[17776]: 77E47604481: to=<[email protected]>, relay=dovecot, delay=0.07, delays=0.02/0.01/0/0.04, dsn=2.0.0, status=sent (delivered via dovecot service (doveconf: Warning: /etc/dovecot/dovecot.conf line 71: Global setting mail_plugins won't change the s)) May 3 14:33:36 anonhost postfix/qmgr[818]: 77E47604481: removed May 3 16:33:37 anonhost amavis[17716]: (17716-18) ip_from_received: no IP address in: from DB9PR02CA0017.eurprd02.prod.outlook.com (2603:10a6:10:1d9::22)\n by VE1P192MB0720.EURP192.PROD.OUTLOOK.COM (2603:10a6:800:14b::7) with\n Microsoft SMTP Server (version=TLS1_2,\n cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4087.35; Mon, 3 May\n 2021 14:33:34 +0000\n May 3 16:33:37 anonhost amavis[17716]: (17716-18) ip_from_received: no IP address in: from DB5EUR01FT032.eop-EUR01.prod.protection.outlook.com\n (2603:10a6:10:1d9:cafe::9f) by DB9PR02CA0017.outlook.office365.com\n (2603:10a6:10:1d9::22) with Microsoft SMTP Server (version=TLS1_2,\n cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4087.27 via Frontend\n Transport; Mon, 3 May 2021 14:33:34 +0000\n
If your client is using his own domain, has he correctly configured SPF? https://o365info.com/how-to-manage-spf-record-in-an-office-365-based-environment/ Discard is bad, because it silently drops the mail. As I understand the log file, its discarded because of a header check. What is the content of the files: /etc/postfix/header_checks /etc/postfix/mime_header_checks /etc/postfix/nested_header_checks By the way: Your main.cf looks like a mess. How do you maintain that? Might be a good time to use the new ISPC way of overwriting the postfix config: https://www.howtoforge.com/communit...-for-custom-postfix-and-dovecot-config.86559/ Then you can go through every line of your config and see if you still need it (do you need IPv4 only? Do you really need TLS1.0/TLS1.1? Do you really want not.therealname.com in mydestination? etc pp ...)
Hiya Stein. Thanks for the reply. As for the ISPC postfig config tool, I first need to upgrade ISPconfig I think and I'm a bit hesitant to do that on a production system... I'm on 3.1.15p3 at the moment. The main.conf is controlled by ISPconfig; I don't touch it myself. By the way "not.therealname.com" is, as the term says, in actuality the FQDN of my server... Below the header_checks file, the others were empty: Code: /^From: .*Mortgage|Concealed|Affiliate|Bathroom|Ukrain|Russia|TopRated|KN95/ DISCARD /^From: .*Be Sure|Dating|Hemp Oil|Debt|Associate|Shooting|Credit Score/ DISCARD /^From: .* Associate|Buy |Benefits|Settlement|Singles|HVAC|Overstock/ DISCARD /^From: .*Body Shape| Bill | Savings| Pain | Miracle | Health |Medicare/ DISCARD /^From: .*Thermo|Fever|Pressure|Cool Air|Lingo|Fast |Explore|[Bb]oost/ DISCARD /^From: .*Doorbell|Smartwatch|CoolAir|Drone|Ultra Wifi|BodyTone|Breathe/ DISCARD /^From: .*Asian|RentTo|Torch|Tactical|Diabet|DIY|Before/ DISCARD /^From: .* Fat |Killer|Gizmos|[Ss]hed |Warby|[Ww]ood|Remed/ DISCARD /^From: .*UltraWifi|Canvas|Prices|[Mm]odeling|Pest |Snoring|Remover|Fungus/ DISCARD I added these to ISPconfig because I had lots of spam issues.. Still have btw, need to find better ways.
Use rspamd agains spam: https://www.howtoforge.com/replacing-amavisd-with-rspamd-in-ispconfig/ (Attention, can lead to some (new) false positives in the beginning, especially if your users get (valid) newsletters) From what I see, one of your header checks is responsible for the discarding of the message. But I don't know, why it works if the message is sent a second time Since you anonymized the FROM part, it is hard to tell which rule might have been applied. You could add an additional message for your log files, for example to see which line is responsible: See: http://www.postfix.org/header_checks.5.html For testing, you could use HOLD instead of DISCARD. So you will see the messages and have the option to delete or release it. Or you can use REJECT instead of DISCARD, so the sender will get a notification and you additional information for the logs. If you have a message that gets reproducibly discarded, you can also use one of the common debugging techniques: http://www.postfix.org/DEBUG_README.html
Your earlier logs show this was the rejected header: So if your header_checks only matches From: headers..... something doesn't add up. In any case, rejecting mail based on very short sequences of characters like that is quite prone to false positives (hence your issue here) and not recommended for most cases. Train your spam scanner and it'll catch a lot more of those; if you need to write custom rules, it would be better to write them in the language/config of your spam scanner than blocking in postfix.
