Debian 11->12 upgrade. Email not sending

The problem still appears to be somewhere in the rspamd configuration. When I try to send a message, nothing useful that I can detect comes out of postfix, and all I get in the debug from rspamd is:

Code:

2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; proxy; proxy_accept_socket: accepted milter connection from 127.0.0.1 port 33928
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_handle_session: read 17 bytes, 0 remain, 65541 allocated
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 0, state: 0
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 1, state: 1
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 2, state: 2
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 3, state: 3
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 4, state: 4
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 5, state: 5
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_process_command: optneg: version: 6, actions: 511, protocol: 2097151
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_send_action: optneg reply: ver=6, actions=511, protocol=1044608
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_handle_session: read 178 bytes, 0 remain, 65541 allocated
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 0, state: 0
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 1, state: 1
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 2, state: 2
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 3, state: 3
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 4, state: 4
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 5, state: 5
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_process_command: got macro command
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_process_command: got macro: j -> f.q.d.n
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_process_command: got macro: {daemon_name} -> f.q.d.n
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_process_command: got macro: {daemon_addr} -> ::1
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_process_command: got macro: v -> Postfix 3.7.10
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_process_command: got macro: _ -> localhost [::1]
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 125, state: 0
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 126, state: 1
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 127, state: 2
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 128, state: 3
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 129, state: 4
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 130, state: 5
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_process_command: got connect command
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_process_command: got hostname on connect phase: localhost
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_process_command: got connection from [::1]:33604
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 147, state: 0
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 148, state: 1
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 149, state: 2
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 150, state: 3
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 151, state: 4
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 152, state: 5
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_process_command: got macro command
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 153, state: 0
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 154, state: 1
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 155, state: 2
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 156, state: 3
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 157, state: 4
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 158, state: 5
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_process_command: got helo command
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_process_command: got helo value: localhost
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 168, state: 0
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 169, state: 1
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 170, state: 2
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 171, state: 3
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 172, state: 4
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 173, state: 5
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_process_command: got abort command
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_session_reset: cleanup common data on abort
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_session_reset: cleanup headers
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 173, state: 0
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 174, state: 1
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 175, state: 2
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 176, state: 3
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_consume_input: offset: 177, state: 4
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_process_command: quit command, refcount: 1
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; proxy; proxy_milter_finish_handler: finished milter connection
2024-02-11 09:39:07 #5089(rspamd_proxy) <da0beb>; milter; rspamd_milter_session_dtor: destroying milter session

Not much to go on!
What other services does rspamd rely on if any? I notice it talks to redis.

 

Here's the mail.log with smtpd -v set in master.cf

Code:

.880140-08:00 *h* postfix/submission/smtpd[17205]: smtp_stream_setup: maxtime=300 enable_deadline=0 min_data_rate=0
.880189-08:00 *h* postfix/submission/smtpd[17205]: match_hostname: smtpd_client_event_limit_exceptions: localhost ~? 127.0.0.0/8
.880243-08:00 *h* postfix/submission/smtpd[17205]: match_hostaddr: smtpd_client_event_limit_exceptions: ::1 ~? 127.0.0.0/8
.880285-08:00 *h* postfix/submission/smtpd[17205]: match_hostname: smtpd_client_event_limit_exceptions: localhost ~? [::1]/128
.880327-08:00 *h* postfix/submission/smtpd[17205]: match_hostaddr: smtpd_client_event_limit_exceptions: ::1 ~? [::1]/128
.880373-08:00 *h* postfix/submission/smtpd[17205]: report connect to all milters
.880419-08:00 *h* postfix/submission/smtpd[17205]: milter8_connect: non-protocol events for protocol version 6:
.880471-08:00 *h* postfix/submission/smtpd[17205]: milter8_connect: transport=inet endpoint=127.0.0.1:11332
.880504-08:00 *h* postfix/submission/smtpd[17205]: trying... [127.0.0.1]
.880549-08:00 *h* postfix/submission/smtpd[17205]: vstream_tweak_tcp: TCP_MAXSEG 32741
.880593-08:00 *h* postfix/submission/smtpd[17205]: fd=22: stream buffer size old=0 new=130964
.880637-08:00 *h* postfix/submission/smtpd[17205]: milter8_connect: my_version=0x6
.880679-08:00 *h* postfix/submission/smtpd[17205]: milter8_connect: my_actions=0x1ff SMFIF_ADDHDRS SMFIF_CHGBODY SMFIF_ADDRCPT SMFIF_DELRCPT SMFIF_CHGHDRS SMFIF_QUARANTINE SMFIF_CHGFROM SMFIF_ADDRCPT_PAR SMFIF_SETSYMLIST
.880716-08:00 *h* postfix/submission/smtpd[17205]: milter8_connect: my_events=0x1fffff SMFIP_NOCONNECT SMFIP_NOHELO SMFIP_NOMAIL SMFIP_NORCPT SMFIP_NOBODY SMFIP_NOHDRS SMFIP_NOEOH SMFIP_NR_HDR SMFIP_NOUNKNOWN SMFIP_NODATA SMFIP_SKIP SMFIP_RCPT_REJ SMFIP_NR_CONN SMFIP_NR_HELO SMFIP_NR_MAIL SMFIP_NR_RCPT SMFIP_NR_DATA SMFIP_NR_UNKN SMFIP_NR_EOH SMFIP_NR_BODY SMFIP_HDR_LEADSPC
.880775-08:00 *h* postfix/submission/smtpd[17205]: milter8_connect: milter inet:127.0.0.1:11332 version 6
.880861-08:00 *h* postfix/submission/smtpd[17205]: milter8_connect: events SMFIP_NR_HDR SMFIP_NR_CONN SMFIP_NR_HELO SMFIP_NR_MAIL SMFIP_NR_RCPT SMFIP_NR_DATA SMFIP_NR_UNKN SMFIP_NR_EOH SMFIP_NR_BODY
.880896-08:00 *h* postfix/submission/smtpd[17205]: milter8_connect: requests SMFIF_ADDHDRS SMFIF_CHGBODY SMFIF_ADDRCPT SMFIF_DELRCPT SMFIF_CHGHDRS SMFIF_QUARANTINE SMFIF_CHGFROM SMFIF_ADDRCPT_PAR SMFIF_SETSYMLIST
.880943-08:00 *h* postfix/submission/smtpd[17205]: milter_macro_lookup: "j"
.880990-08:00 *h* postfix/submission/smtpd[17205]: milter_macro_lookup: result "*f.q.d.n*"
.881027-08:00 *h* postfix/submission/smtpd[17205]: milter_macro_lookup: "{daemon_name}"
.881081-08:00 *h* postfix/submission/smtpd[17205]: milter_macro_lookup: result "*f.q.d.n*"
.881118-08:00 *h* postfix/submission/smtpd[17205]: milter_macro_lookup: "{daemon_addr}"
.881166-08:00 *h* postfix/submission/smtpd[17205]: milter_macro_lookup: result "::1"
.881207-08:00 *h* postfix/submission/smtpd[17205]: milter_macro_lookup: "v"
.881253-08:00 *h* postfix/submission/smtpd[17205]: milter_macro_lookup: result "Postfix 3.7.10"
.881302-08:00 *h* postfix/submission/smtpd[17205]: milter_macro_lookup: "_"
.881336-08:00 *h* postfix/submission/smtpd[17205]: milter_macro_lookup: result "localhost [::1]"
.881377-08:00 *h* postfix/submission/smtpd[17205]: milter8_conn_event: milter inet:127.0.0.1:11332: connect localhost/::1
.881426-08:00 *h* postfix/submission/smtpd[17205]: event: SMFIC_CONNECT; macros: j=*f.q.d.n* {daemon_name}=*f.q.d.n* {daemon_addr}=::1 v=Postfix 3.7.10 _=localhost [::1]
.881468-08:00 *h* postfix/submission/smtpd[17205]: skipping reply for event SMFIC_CONNECT from milter inet:127.0.0.1:11332
.881514-08:00 *h* postfix/submission/smtpd[17205]: > localhost[::1]: 220 *f.q.d.n* ESMTP Postfix (Debian/GNU)
.881569-08:00 *h* postfix/submission/smtpd[17205]: smtp_stream_setup: maxtime=300 enable_deadline=0 min_data_rate=0
.881610-08:00 *h* postfix/submission/smtpd[17205]: < localhost[::1]: EHLO localhost
.881662-08:00 *h* postfix/submission/smtpd[17205]: report helo to all milters
.881704-08:00 *h* postfix/submission/smtpd[17205]: milter_macro_lookup: "{tls_version}"
.881749-08:00 *h* postfix/submission/smtpd[17205]: milter_macro_lookup: "{cipher}"
.881795-08:00 *h* postfix/submission/smtpd[17205]: milter_macro_lookup: "{cipher_bits}"
.881832-08:00 *h* postfix/submission/smtpd[17205]: milter_macro_lookup: "{cert_subject}"
.881876-08:00 *h* postfix/submission/smtpd[17205]: milter_macro_lookup: "{cert_issuer}"
.881919-08:00 *h* postfix/submission/smtpd[17205]: milter8_helo_event: milter inet:127.0.0.1:11332: helo localhost
.881963-08:00 *h* postfix/submission/smtpd[17205]: event: SMFIC_HELO; macros: (none)
.882001-08:00 *h* postfix/submission/smtpd[17205]: skipping reply for event SMFIC_HELO from milter inet:127.0.0.1:11332
.882044-08:00 *h* postfix/submission/smtpd[17205]: match_list_match: localhost: no match
.882091-08:00 *h* postfix/submission/smtpd[17205]: match_list_match: ::1: no match
.882131-08:00 *h* postfix/submission/smtpd[17205]: > localhost[::1]: 250-*f.q.d.n*
.882179-08:00 *h* postfix/submission/smtpd[17205]: > localhost[::1]: 250-PIPELINING
.882226-08:00 *h* postfix/submission/smtpd[17205]: > localhost[::1]: 250-SIZE
.882270-08:00 *h* postfix/submission/smtpd[17205]: > localhost[::1]: 250-VRFY
.882312-08:00 *h* postfix/submission/smtpd[17205]: > localhost[::1]: 250-ETRN
.882356-08:00 *h* postfix/submission/smtpd[17205]: > localhost[::1]: 250-STARTTLS
.882403-08:00 *h* postfix/submission/smtpd[17205]: > localhost[::1]: 250-ENHANCEDSTATUSCODES
.882445-08:00 *h* postfix/submission/smtpd[17205]: > localhost[::1]: 250-8BITMIME
.882494-08:00 *h* postfix/submission/smtpd[17205]: > localhost[::1]: 250-DSN
.882537-08:00 *h* postfix/submission/smtpd[17205]: > localhost[::1]: 250 CHUNKING
.882584-08:00 *h* postfix/submission/smtpd[17205]: smtp_stream_setup: maxtime=300 enable_deadline=0 min_data_rate=0
.882618-08:00 *h* postfix/submission/smtpd[17205]: < localhost[::1]: QUIT
.882662-08:00 *h* postfix/submission/smtpd[17205]: > localhost[::1]: 221 2.0.0 Bye
.882714-08:00 *h* postfix/submission/smtpd[17205]: match_hostname: smtpd_client_event_limit_exceptions: localhost ~? 127.0.0.0/8
.882752-08:00 *h* postfix/submission/smtpd[17205]: match_hostaddr: smtpd_client_event_limit_exceptions: ::1 ~? 127.0.0.0/8
.882793-08:00 *h* postfix/submission/smtpd[17205]: match_hostname: smtpd_client_event_limit_exceptions: localhost ~? [::1]/128
.882832-08:00 *h* postfix/submission/smtpd[17205]: match_hostaddr: smtpd_client_event_limit_exceptions: ::1 ~? [::1]/128
.882878-08:00 *h* postfix/submission/smtpd[17205]: abort all milters
.882916-08:00 *h* postfix/submission/smtpd[17205]: milter8_abort: abort milter inet:127.0.0.1:11332
.882964-08:00 *h* postfix/submission/smtpd[17205]: disconnect event to all milters
.883010-08:00 *h* postfix/submission/smtpd[17205]: milter8_disc_event: quit milter inet:127.0.0.1:11332
.883047-08:00 *h* postfix/submission/smtpd[17205]: disconnect from localhost[::1] ehlo=1 quit=1 commands=2
.883094-08:00 *h* postfix/submission/smtpd[17205]: free all milters
.883135-08:00 *h* postfix/submission/smtpd[17205]: free milter inet:127.0.0.1:11332
.883185-08:00 *h* postfix/submission/smtpd[17205]: master_notify: status 1
.883228-08:00 *h* postfix/submission/smtpd[17205]: connection closed
.866706-08:00 *h* postfix/submission/smtpd[17205]: proxymap stream disconnect
.866824-08:00 *h* postfix/submission/smtpd[17205]: auto_clnt_close: disconnect private/tlsmgr stream

This appears to be the handoff to the milter where it fails.

 

I think (hope) I'm getting closer.
I believe I resolved the issue with rspamd as there was a mismatch in the protocol number being used (postfix was 2 and rspamd wanted 6). I am no longer getting errors from rspamd or the milter.
The error now seems to be in the connection between round cube and the smtp server. After getting debug on in round cube (and having it log to syslog as it didn't seem to want to log to /var/log/roundcube) I finally see a real error (hopefully)

Code:

: <2k1dpqlt> Recv: 250-PIPELINING
2024-02-11T21:57:54.052821-08:00 s91370 roundcube: <2k1dpqlt> Recv: 250-SIZE
2024-02-11T21:57:54.052852-08:00 s91370 roundcube: <2k1dpqlt> Recv: 250-VRFY
2024-02-11T21:57:54.052881-08:00 s91370 roundcube: <2k1dpqlt> Recv: 250-ETRN
2024-02-11T21:57:54.052911-08:00 s91370 roundcube: <2k1dpqlt> Recv: 250-STARTTLS
2024-02-11T21:57:54.053012-08:00 s91370 roundcube: <2k1dpqlt> Recv: 250-ENHANCEDSTATUSCODES
2024-02-11T21:57:54.053046-08:00 s91370 roundcube: <2k1dpqlt> Recv: 250-8BITMIME
2024-02-11T21:57:54.053090-08:00 s91370 roundcube: <2k1dpqlt> Recv: 250-DSN
2024-02-11T21:57:54.053130-08:00 s91370 roundcube: <2k1dpqlt> Recv: 250 CHUNKING
2024-02-11T21:57:54.053172-08:00 s91370 roundcube: PHP Deprecated:  Creation of dynamic property PEAR_Error::$callback is deprecated in /usr/share/php/PEAR.php on line 905
2024-02-11T21:57:54.053207-08:00 s91370 roundcube: <2k1dpqlt> PHP Error: SMTP server does not support authentication (POST /squirrelmail/?_task=mail&_unlock=loading1707717473900&_framed=1&_action=send)
2024-02-11T21:57:54.053261-08:00 s91370 roundcube: <2k1dpqlt> Send: QUIT
2024-02-11T21:57:54.054091-08:00 s91370 roundcube: <2k1dpqlt> Recv: 221 2.0.0 Bye
2024-02-11T21:57:54.054138-08:00 s91370 roundcube: <2k1dpqlt> SMTP Error: Authentication failure: f.q.d.n

my roundcube config.inc.php parameters are:

Code:

$config['default_host'] = 'localhost';
$config['smtp_server'] = 'localhost';
$config['smtp_port'] = 587;
$config['smtp_user'] = '%u';
$config['smtp_pass'] = '%p';
$config['support_url'] = '';
$config['product_name'] = 'Roundcube Webmail';
$config['des_key'] = 'removed for privacy';
$config['plugins'] = array(
);

I believe this is coming down to another certificate problem with SSL/STARTTLS

 

Here it is:

Code:

#
# Postfix master process configuration file.  For details on the format
# of the file, see the master(5) manual page (command: "man 5 master" or
# on-line: http://www.postfix.org/master.5.html).
#
# Do not forget to execute "postfix reload" after editing this file.
#
# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (no)    (never) (100)
# ==========================================================================
smtp      inet  n       -       y       -       -       smtpd
#smtp      inet  n       -       y       -       1       postscreen
#smtpd     pass  -       -       y       -       -       smtpd
#dnsblog   unix  -       -       y       -       0       dnsblog
#tlsproxy  unix  -       -       y       -       0       tlsproxy
submission inet n       -       y       -       -       smtpd -v
 -o syslog_name=postfix/submission
 -o smtpd_tls_security_level=encrypt
 -o smtpd_sasl_auth_enable=yes
 -o smtpd_client_restrictions=permit_sasl_authenticated,reject
#  -o syslog_name=postfix/submission
#  -o smtpd_tls_security_level=encrypt
#  -o smtpd_sasl_auth_enable=yes
#  -o smtpd_tls_auth_only=yes
#  -o smtpd_reject_unlisted_recipient=no
#  -o smtpd_client_restrictions=$mua_client_restrictions
#  -o smtpd_helo_restrictions=$mua_helo_restrictions
#  -o smtpd_sender_restrictions=$mua_sender_restrictions
#  -o smtpd_recipient_restrictions=
#  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
#  -o milter_macro_daemon_name=ORIGINATING
smtps     inet  n       -       y       -       -       smtpd
 -o syslog_name=postfix/smtps
 -o smtpd_tls_wrappermode=yes
 -o smtpd_sasl_auth_enable=yes
 -o smtpd_client_restrictions=permit_sasl_authenticated,reject
#  -o syslog_name=postfix/smtps
#  -o smtpd_tls_wrappermode=yes
#  -o smtpd_sasl_auth_enable=yes
#  -o smtpd_reject_unlisted_recipient=no
#  -o smtpd_client_restrictions=$mua_client_restrictions
#  -o smtpd_helo_restrictions=$mua_helo_restrictions
#  -o smtpd_sender_restrictions=$mua_sender_restrictions
#  -o smtpd_recipient_restrictions=
#  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
#  -o milter_macro_daemon_name=ORIGINATING
#628       inet  n       -       y       -       -       qmqpd
pickup    unix  n       -       y       60      1       pickup
cleanup   unix  n       -       y       -       0       cleanup
qmgr      unix  n       -       n       300     1       qmgr
#qmgr     unix  n       -       n       300     1       oqmgr
tlsmgr    unix  -       -       y       1000?   1       tlsmgr
rewrite   unix  -       -       y       -       -       trivial-rewrite
bounce    unix  -       -       y       -       0       bounce
defer     unix  -       -       y       -       0       bounce
trace     unix  -       -       y       -       0       bounce
verify    unix  -       -       y       -       1       verify
flush     unix  n       -       y       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
proxywrite unix -       -       n       -       1       proxymap
smtp      unix  -       -       y       -       -       smtp
relay     unix  -       -       y       -       -       smtp
        -o syslog_name=postfix/$service_name
#       -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
showq     unix  n       -       y       -       -       showq
error     unix  -       -       y       -       -       error
retry     unix  -       -       y       -       -       error
discard   unix  -       -       y       -       -       discard
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual
lmtp      unix  -       -       y       -       -       lmtp
anvil     unix  -       -       y       -       1       anvil
scache    unix  -       -       y       -       1       scache
postlog   unix-dgram n  -       n       -       1       postlogd
#
# ====================================================================
# Interfaces to non-Postfix software. Be sure to examine the manual
# pages of the non-Postfix software to find out what options it wants.
#
# Many of the following services use the Postfix pipe(8) delivery
# agent.  See the pipe(8) man page for information about ${recipient}
# and other message envelope options.
# ====================================================================
#
# maildrop. See the Postfix MAILDROP_README file for details.
# Also specify in main.cf: maildrop_destination_recipient_limit=1
#
maildrop  unix  -       n       n       -       -       pipe
  flags=DRXhu user=vmail argv=/usr/bin/maildrop -d ${recipient}
#
# ====================================================================
#
# Recent Cyrus versions can use the existing "lmtp" master.cf entry.
#
# Specify in cyrus.conf:
#   lmtp    cmd="lmtpd -a" listen="localhost:lmtp" proto=tcp4
#
# Specify in main.cf one or more of the following:
#  mailbox_transport = lmtp:inet:localhost
#  virtual_transport = lmtp:inet:localhost
#
# ====================================================================
#
# Cyrus 2.1.5 (Amos Gouaux)
# Also specify in main.cf: cyrus_destination_recipient_limit=1
#
#cyrus     unix  -       n       n       -       -       pipe
#  flags=DRX user=cyrus argv=/cyrus/bin/deliver -e -r ${sender} -m ${extension} ${user}
#
# ====================================================================
# Old example of delivery via Cyrus.
#
#old-cyrus unix  -       n       n       -       -       pipe
#  flags=R user=cyrus argv=/cyrus/bin/deliver -e -m ${extension} ${user}
#
# ====================================================================
#
# See the Postfix UUCP_README file for configuration details.
#
uucp      unix  -       n       n       -       -       pipe
  flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
#
# Other external delivery methods.
#
ifmail    unix  -       n       n       -       -       pipe
  flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
bsmtp     unix  -       n       n       -       -       pipe
  flags=Fq. user=bsmtp argv=/usr/lib/bsmtp/bsmtp -t$nexthop -f$sender $recipient
scalemail-backend unix -       n       n       -       2       pipe
  flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store ${nexthop} ${user} ${extension}
mailman   unix  -       n       n       -       -       pipe
  flags=FRX user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py ${nexthop} ${user}
dovecot   unix  -       n       n       -       -       pipe
  flags=DRhu user=vmail:vmail argv=/usr/lib/dovecot/deliver -f ${sender} -d ${user}@${nexthop}

I also ran https://testssl.sh/ on the subject domain and the only errors it comes back with is a self signed certificate and mismatched URI to host.

Code:

[B] Common Name (CN)             [/B][I]f.q.d.n[/I]
[B] subjectAltName (SAN)         [/B]missing -- no SAN is deprecated
[B] Trust (hostname)             [/B]certificate does not match supplied URI (same w/o SNI)
[B] Chain of trust[/B]               [B]NOT ok[/B] (self signed)
[B] EV cert[/B] (experimental)       no
[B] Certificate Validity (UTC)   [/B]3647 >= 60 days (2024-02-09 20:41 --> 2034-02-06 20:41)
                              >= 10 years is way too long
[B] ETS/"eTLS"[/B], visibility info  not present
[B] Certificate Revocation List  [/B]--
[B] OCSP URI                     [/B]--
                             NOT ok -- neither CRL nor OCSP URI provided
[B] OCSP stapling                [/B]not offered
[B] OCSP must staple extension   [/B]--
[B] DNS CAA RR[/B] (experimental)    [B]not offered
 Certificate Transparency     [/B]N/A
[B] Certificates provided[/B]        1
[B] Issuer                       [/B][I]f.q.d.n[/I] ([I]Windward Systems[/I] from [I]CA[/I])
[B] Intermediate Bad OCSP[/B] (exp.) Ok

 

i really appreciate you looking at it. I am off to bed but I will pick this up in the morning. I feel like we are close, so I am loathe to give up and reinstall. It's not in my nature

 

Well, that said, it is all moot now. It turns out the server needs to be reinstalled for completely unrelated reasons anyway. So that's where this story ends. I will be starting fresh with Debian 12 so hopefully that will completely avoid any potential issues from upgrading that occurred last time.
Thanks again for your help Till. I always learn the most from troubleshooting and I already feel like I have a better handle on ISPConfig and its related services. Hopefully this thread helps to point someone else in the right direction and maybe we'll find out what the issue was some other time.
Cheers

P.S. After a full reinstall of Debian 12 (clean, no upgrade) I reinstalled ISPConfig and there were no issues with sending mail. So clearly something was lost/messed up in the upgrade last time.