Multiserver - mx1 and mx2 Rspamd GUI issue

Hi All,
Apologies for this really long post, but there is a lot of info and tests I've already gone through.

I have 2 multiserver systems, 1 Dev located locally and 1 Prod located on hosted hardware. Both systems have the same ISPConfig servers (1 x panel, 2 x web, 2 x mx and 2 x ns).
The Dev system works perfectly.

The issue is with the Prod system. Panel, web1, web2, ns1 and ns2 are working. The issue is with mx1 and mx2.
So I've followed the entire 'ispconfig-multiserver-setup-debian-ubuntu' to complete the setup, but revisited these 2 pages multiple times to reinstall mx1 and mx2 for one reason or another
https://www.howtoforge.com/tutorial/ispconfig-multiserver-setup-debian-ubuntu/3/
and
https://www.howtoforge.com/tutorial/ispconfig-multiserver-setup-debian-ubuntu/4/
except on the latest install I left off the '--use-unbound' parameter to resolve an issue. I have installed these servers multiple times but still have this Rspamd GUI issue.
I can't access "https://mx1.example.com/rspamd/" or "https://mx2.example.com/rspamd/". I have spent 3 days looking at this and am not sure where to go from here.
The DNS records for mx1.example.com point to the external firewallIP:123.123.123.22, that has a 1:1 entry in my firewall to internal serverIP:192.168.1.101. My IP address here where I access my Prod system is whitelisted in the firewall so there should be no blocking there. But I have still created rules for 80 and 443.
I can see from the firewall logs that myip:443 traffic hits the firewall on externalIP:123.123.123.22 and is being allowed/passed onto the servers internalIP: 192.168.1.101:443 but if I look at my Apache logs "/var/log/apache/access.log" and "/var/log/apache/error.log" I don't see anything.
I have observed the following on my Prod system:

Code:

http://mx1.example.com -> default Apache webpage on mx1
https://mx1.example.com -> timeout
https://mx1.example.com/rspamd/ -> timeout

On my Development system:

Code:

http://mx1.example.com -> default Apache webpage on mx1
https://mx1.example.com -> default ISPConfig web template (/var/www/mx1.example.com/web/)
https://mx1.example.com/rspamd/ -> Rspamd GUI

I have compared the Apache files/folders (conf-enable/mods-enable/apache2.conf/envvars/magic/ports.conf) and they are all exactly the same and the sites-enabled is exactly the same apart from obvious differences for domain names (mx1.example.com) and document paths (/var/www/mx1.example.com/).

I have run your Test Script from 'https://forum.howtoforge.com/threads/please-read-before-posting.58408/' and all the services are running.

Code:

cat htf_report.txt
##### SERVER #####
IP-address (as per hostname): ***.***.***.***
[WARN] could not determine server's ip address by ifconfig
[INFO] OS version is Debian GNU/Linux 12 (bookworm)
 
[INFO] uptime:  13:02:34 up 11:10,  1 user,  load average: 0.01, 0.02, 0.00
 
[INFO] memory:
               total        used        free      shared  buff/cache   available
Mem:           7.8Gi       2.3Gi       4.9Gi        26Mi       841Mi       5.5Gi
Swap:          952Mi          0B       952Mi
 
[INFO] systemd failed services status:
  UNIT LOAD ACTIVE SUB DESCRIPTION
0 loaded units listed.

[INFO] ISPConfig is installed.

##### ISPCONFIG #####
ISPConfig version is 3.2.12p1


##### VERSION CHECK #####

[INFO] php (cli) version is 8.2.26
[INFO] php-cgi (used for cgi php in default vhost!) is version 8.2.26

##### PORT CHECK #####

[WARN] Port 8080 (ISPConfig) seems NOT to be listening

##### MAIL SERVER CHECK #####

[WARN] I found no "smtps" entry in your postfix master.cf
[INFO] this is not critical, but if you want to offer SSL for smtp (not TLS) connections you have to enable this.

##### RUNNING SERVER PROCESSES #####

[INFO] I found the following web server(s):
        Apache 2 (PID 40818)
[INFO] I found the following mail server(s):
        Postfix (PID 1336)
[INFO] I found the following pop3 server(s):
        Dovecot (PID 637)
[INFO] I found the following imap server(s):
        Dovecot (PID 637)
[INFO] I found the following ftp server(s):
        PureFTP (PID 1063)

##### LISTENING PORTS #####
(only           ()
Local           (Address)
[localhost]:6379                (650/redis-server)
***.***.***.***:53              (643/named)
***.***.***.***:53              (643/named)
[anywhere]:3306         (837/mariadbd)
[localhost]:11211               (642/memcached)
[localhost]:11334               (691/rspamd:)
[localhost]:11333               (691/rspamd:)
[localhost]:11332               (691/rspamd:)
[localhost]:53          (643/named)
[localhost]:53          (643/named)
[localhost]:953         (643/named)
[localhost]:953         (643/named)
[anywhere]:993          (637/dovecot)
[anywhere]:995          (637/dovecot)
[anywhere]:587          (1336/master)
[anywhere]:465          (1336/master)
[anywhere]:143          (637/dovecot)
[localhost]:10023               (647/postgrey)
[anywhere]:12345                (637/dovecot)
[anywhere]:21           (1063/pure-ftpd)
[anywhere]:22           (682/sshd:)
[anywhere]:25           (1336/master)
[anywhere]:110          (637/dovecot)
[anywhere]:4190         (637/dovecot)
*:*:*:*::*:8081         (40818/apache2)
*:*:*:*::*:10023                (647/postgrey)
*:*:*:*::*:3306         (837/mariadbd)
*:*:*:*::*:953          (643/named)
*:*:*:*::*:953          (643/named)
*:*:*:*::*be24:11ff:febb:53             (643/named)
*:*:*:*::*be24:11ff:febb:53             (643/named)
*:*:*:*::*:53           (643/named)
*:*:*:*::*:53           (643/named)
*:*:*:*::*:11333                (691/rspamd:)
*:*:*:*::*:11332                (691/rspamd:)
*:*:*:*::*:11334                (691/rspamd:)
*:*:*:*::*:993          (637/dovecot)
*:*:*:*::*:995          (637/dovecot)
*:*:*:*::*:587          (1336/master)
*:*:*:*::*:443          (40818/apache2)
*:*:*:*::*:6379         (650/redis-server)
*:*:*:*::*:465          (1336/master)
[localhost]43           (637/dovecot)
[localhost]2345         (637/dovecot)
*:*:*:*::*:21           (1063/pure-ftpd)
*:*:*:*::*:22           (682/sshd:)
*:*:*:*::*:25           (1336/master)
[localhost]10           (637/dovecot)
*:*:*:*::*:80           (40818/apache2)
*:*:*:*::*:4190         (637/dovecot)




##### IPTABLES #####
Chain INPUT (policy DROP)
target     prot opt source               destination         
f2b-postfix-sasl  6    --  [anywhere]/0            [anywhere]/0            multiport dports 25
ufw-before-logging-input  0    --  [anywhere]/0            [anywhere]/0           
ufw-before-input  0    --  [anywhere]/0            [anywhere]/0           
ufw-after-input  0    --  [anywhere]/0            [anywhere]/0           
ufw-after-logging-input  0    --  [anywhere]/0            [anywhere]/0           
ufw-reject-input  0    --  [anywhere]/0            [anywhere]/0           
ufw-track-input  0    --  [anywhere]/0            [anywhere]/0           

Chain FORWARD (policy DROP)
target     prot opt source               destination         
ufw-before-logging-forward  0    --  [anywhere]/0            [anywhere]/0           
ufw-before-forward  0    --  [anywhere]/0            [anywhere]/0           
ufw-after-forward  0    --  [anywhere]/0            [anywhere]/0           
ufw-after-logging-forward  0    --  [anywhere]/0            [anywhere]/0           
ufw-reject-forward  0    --  [anywhere]/0            [anywhere]/0           
ufw-track-forward  0    --  [anywhere]/0            [anywhere]/0           

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ufw-before-logging-output  0    --  [anywhere]/0            [anywhere]/0           
ufw-before-output  0    --  [anywhere]/0            [anywhere]/0           
ufw-after-output  0    --  [anywhere]/0            [anywhere]/0           
ufw-after-logging-output  0    --  [anywhere]/0            [anywhere]/0           
ufw-reject-output  0    --  [anywhere]/0            [anywhere]/0           
ufw-track-output  0    --  [anywhere]/0            [anywhere]/0           

Chain f2b-postfix-sasl (1 references)
target     prot opt source               destination         
REJECT     0    --  ***.***.***.***       [anywhere]/0            reject-with icmp-port-unreachable
RETURN     0    --  [anywhere]/0            [anywhere]/0           

Chain ufw-after-forward (1 references)
target     prot opt source               destination         

Chain ufw-after-input (1 references)
target     prot opt source               destination         
ufw-skip-to-policy-input  17   --  [anywhere]/0            [anywhere]/0            udp dpt:137
ufw-skip-to-policy-input  17   --  [anywhere]/0            [anywhere]/0            udp dpt:138
ufw-skip-to-policy-input  6    --  [anywhere]/0            [anywhere]/0            tcp dpt:139
ufw-skip-to-policy-input  6    --  [anywhere]/0            [anywhere]/0            tcp dpt:445
ufw-skip-to-policy-input  17   --  [anywhere]/0            [anywhere]/0            udp dpt:67
ufw-skip-to-policy-input  17   --  [anywhere]/0            [anywhere]/0            udp dpt:68
ufw-skip-to-policy-input  0    --  [anywhere]/0            [anywhere]/0            ADDRTYPE match dst-type BROADCAST

Chain ufw-after-logging-forward (1 references)
target     prot opt source               destination         
LOG        0    --  [anywhere]/0            [anywhere]/0            limit: avg 3/min burst 10 LOG flags 0 level 4 prefix "[UFW BLOCK] "

Chain ufw-after-logging-input (1 references)
target     prot opt source               destination         
LOG        0    --  [anywhere]/0            [anywhere]/0            limit: avg 3/min burst 10 LOG flags 0 level 4 prefix "[UFW BLOCK] "

Chain ufw-after-logging-output (1 references)
target     prot opt source               destination         

Chain ufw-after-output (1 references)
target     prot opt source               destination         

Chain ufw-before-forward (1 references)
target     prot opt source               destination         
ACCEPT     0    --  [anywhere]/0            [anywhere]/0            ctstate RELATED,ESTABLISHED
ACCEPT     1    --  [anywhere]/0            [anywhere]/0            icmptype 3
ACCEPT     1    --  [anywhere]/0            [anywhere]/0            icmptype 11
ACCEPT     1    --  [anywhere]/0            [anywhere]/0            icmptype 12
ACCEPT     1    --  [anywhere]/0            [anywhere]/0            icmptype 8
ufw-user-forward  0    --  [anywhere]/0            [anywhere]/0           

Chain ufw-before-input (1 references)
target     prot opt source               destination         
ACCEPT     0    --  [anywhere]/0            [anywhere]/0           
ACCEPT     0    --  [anywhere]/0            [anywhere]/0            ctstate RELATED,ESTABLISHED
ufw-logging-deny  0    --  [anywhere]/0            [anywhere]/0            ctstate INVALID
DROP       0    --  [anywhere]/0            [anywhere]/0            ctstate INVALID
ACCEPT     1    --  [anywhere]/0            [anywhere]/0            icmptype 3
ACCEPT     1    --  [anywhere]/0            [anywhere]/0            icmptype 11
ACCEPT     1    --  [anywhere]/0            [anywhere]/0            icmptype 12
ACCEPT     1    --  [anywhere]/0            [anywhere]/0            icmptype 8
ACCEPT     17   --  [anywhere]/0            [anywhere]/0            udp spt:67 dpt:68
ufw-not-local  0    --  [anywhere]/0            [anywhere]/0           
ACCEPT     17   --  [anywhere]/0            ***.***.***.***          udp dpt:5353
ACCEPT     17   --  [anywhere]/0            ***.***.***.***      udp dpt:1900
ufw-user-input  0    --  [anywhere]/0            [anywhere]/0           

Chain ufw-before-logging-forward (1 references)
target     prot opt source               destination         

Chain ufw-before-logging-input (1 references)
target     prot opt source               destination         

Chain ufw-before-logging-output (1 references)
target     prot opt source               destination         

Chain ufw-before-output (1 references)
target     prot opt source               destination         
ACCEPT     0    --  [anywhere]/0            [anywhere]/0           
ACCEPT     0    --  [anywhere]/0            [anywhere]/0            ctstate RELATED,ESTABLISHED
ufw-user-output  0    --  [anywhere]/0            [anywhere]/0           

Chain ufw-logging-allow (0 references)
target     prot opt source               destination         
LOG        0    --  [anywhere]/0            [anywhere]/0            limit: avg 3/min burst 10 LOG flags 0 level 4 prefix "[UFW ALLOW] "

Chain ufw-logging-deny (2 references)
target     prot opt source               destination         
RETURN     0    --  [anywhere]/0            [anywhere]/0            ctstate INVALID limit: avg 3/min burst 10
LOG        0    --  [anywhere]/0            [anywhere]/0            limit: avg 3/min burst 10 LOG flags 0 level 4 prefix "[UFW BLOCK] "

Chain ufw-not-local (1 references)
target     prot opt source               destination         
RETURN     0    --  [anywhere]/0            [anywhere]/0            ADDRTYPE match dst-type LOCAL
RETURN     0    --  [anywhere]/0            [anywhere]/0            ADDRTYPE match dst-type MULTICAST
RETURN     0    --  [anywhere]/0            [anywhere]/0            ADDRTYPE match dst-type BROADCAST
ufw-logging-deny  0    --  [anywhere]/0            [anywhere]/0            limit: avg 3/min burst 10
DROP       0    --  [anywhere]/0            [anywhere]/0           

Chain ufw-reject-forward (1 references)
target     prot opt source               destination         

Chain ufw-reject-input (1 references)
target     prot opt source               destination         

Chain ufw-reject-output (1 references)
target     prot opt source               destination         

Chain ufw-skip-to-policy-forward (0 references)
target     prot opt source               destination         
DROP       0    --  [anywhere]/0            [anywhere]/0           

Chain ufw-skip-to-policy-input (7 references)
target     prot opt source               destination         
DROP       0    --  [anywhere]/0            [anywhere]/0           

Chain ufw-skip-to-policy-output (0 references)
target     prot opt source               destination         
ACCEPT     0    --  [anywhere]/0            [anywhere]/0           

Chain ufw-track-forward (1 references)
target     prot opt source               destination         

Chain ufw-track-input (1 references)
target     prot opt source               destination         

Chain ufw-track-output (1 references)
target     prot opt source               destination         
ACCEPT     6    --  [anywhere]/0            [anywhere]/0            ctstate NEW
ACCEPT     17   --  [anywhere]/0            [anywhere]/0            ctstate NEW

Chain ufw-user-forward (1 references)
target     prot opt source               destination         

Chain ufw-user-input (1 references)
target     prot opt source               destination         
ACCEPT     6    --  [anywhere]/0            [anywhere]/0            tcp dpt:22
ACCEPT     6    --  [anywhere]/0            [anywhere]/0            tcp dpt:25
ACCEPT     6    --  [anywhere]/0            [anywhere]/0            tcp dpt:80
ACCEPT     6    --  [anywhere]/0            [anywhere]/0            tcp dpt:110
ACCEPT     6    --  [anywhere]/0            [anywhere]/0            tcp dpt:143
ACCEPT     6    --  [anywhere]/0            [anywhere]/0            tcp dpt:443
ACCEPT     6    --  [anywhere]/0            [anywhere]/0            tcp dpt:465
ACCEPT     6    --  [anywhere]/0            [anywhere]/0            tcp dpt:587
ACCEPT     6    --  [anywhere]/0            [anywhere]/0            tcp dpt:993
ACCEPT     6    --  [anywhere]/0            [anywhere]/0            tcp dpt:995
ACCEPT     6    --  ***.***.***.***/24       [anywhere]/0            tcp dpt:3306
ACCEPT     6    --  ***.***.***.***/24       [anywhere]/0            tcp dpt:12345

Chain ufw-user-limit (0 references)
target     prot opt source               destination         
LOG        0    --  [anywhere]/0            [anywhere]/0            limit: avg 3/min burst 5 LOG flags 0 level 4 prefix "[UFW LIMIT BLOCK] "
REJECT     0    --  [anywhere]/0            [anywhere]/0            reject-with icmp-port-unreachable

Chain ufw-user-limit-accept (0 references)
target     prot opt source               destination         
ACCEPT     0    --  [anywhere]/0            [anywhere]/0           

Chain ufw-user-logging-forward (0 references)
target     prot opt source               destination         

Chain ufw-user-logging-input (0 references)
target     prot opt source               destination         

Chain ufw-user-logging-output (0 references)
target     prot opt source               destination         

Chain ufw-user-output (1 references)
target     prot opt source               destination         




##### LET'S ENCRYPT #####
acme.sh is installed in /root/.acme.sh/acme.sh

Both Apache and Rspamd are running.

It feels like port 443 isn't running but it is. Or the firewall offloads the 443 packets to the server but the server never receives them.
Could anyone point me in a direction where to perform any further investigation/testing or has seen this before? Maybe someone could shine some light on how the app/8081 works in ISPConfig to deliver the Rspamd GUI on standard 443.

Many Thanks for any help or guidance with this.

 

Hi Till,
Thank you for you quick reply.

Sorry, forgot to say, I'm on Debian 12 (Bookworm) on both Dev and Prod systems. Trying to keep them as identical as possible for this very purpose.
Dev: Debian 12.7 || Kernel: 6.1.0-26-amd64 || Rspamd: 3.10.1
Prod: Debian 12.9 || Kernel: 6.1.0-31-amd64 || Rspamd: 3.11.0

I ruled out there being an issue with Rspamd by me not being able to access the template website at 'https://mx1.example.com'. Is that not the case? Shouldn't I still see be able to see that or at least a request hit my Apache access.log for 443? (for both https://mx1.example.com and https://mx1.example.com/rspamd/) even though they failed.

I'll try a downgrade on Prod to the version that works on Dev (3.10.1) and see if that makes a difference.

 

Hi,

Rspamd refuses to downgrade:

Code:

root@mx1:~# apt install rspamd=3.10.1-1~9de95c2b6~bookworm
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package rspamd is a virtual package provided by:
  rspamd-asan 3.11.0-1~90a175b45~bookworm
You should explicitly select one to install.

E: Version '3.10.1-1~9de95c2b6~bookworm' for 'rspamd' was not found

It would appear I'm already on the latest package...
Dev:

Code:

root@mx1:~# apt list rspamd -a
Listing... Done
rspamd/unknown 3.11.0-1~90a175b45~bookworm amd64 [upgradable from: 3.10.1-1~9de95c2b6~bookworm]
rspamd/now 3.10.1-1~9de95c2b6~bookworm amd64 [installed,upgradable to: 3.11.0-1~90a175b45~bookworm]
rspamd/stable 3.4-1 amd64

Prod:

Code:

root@mx1:~# apt list rspamd -a
Listing... Done
rspamd/unknown,now 3.11.0-1~90a175b45~bookworm amd64 [installed]
rspamd/stable 3.4-1 amd64

I have the .deb file on my Dev system so I could try downgrading with that...

 

Hi Till,
Sorry, I've tried that too. I still get the 'ERR_CONNECTION_TIMED_OUT' response.

 

Snippet from the htf_report.txt file above:

Code:

[localhost]:11334               (691/rspamd:)
[localhost]:11333               (691/rspamd:)
[localhost]:11332               (691/rspamd:)

It feels like it could be a firewall issue, but I see in the logs that myLocalIP is allowed/passed onto serverInternalIP:443 and from the above question, myLocalIP is allowed/passed onto serverInternalIP:8081. So the firewall is allowing/directing the traffic onto the server. It just seems like it isn't getting there.
At a minimum should I be able to access https://mx1.example.com and see the ISPConfig template website? That would rule out an issue with Rspamd and make it a HTTPS/443 port issue.

 

Good call.

Code:

root@mx1@~# wget https://mx1.example.com/rspamd
--2025-02-20 15:21:38--  https://mx1.example.com/rspamd
Resolving mx1.example.com (mx1.example.com)... 127.0.1.1, 192.168.1.101
Connecting to mx1.example.com (mx1.example.com)|127.0.1.1|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://mx1.example.com/rspamd/ [following]
--2025-02-20 15:21:44--  https://mx1.example.com/rspamd/
Reusing existing connection to mx1.example.com:443.
HTTP request sent, awaiting response... 200 OK
Length: 35180 (34K) [text/html]
Saving to: rspamd’

rspamd                                100%[=========================================================================>]  34.36K  --.-KB/s    in 0.002s 

2025-02-20 15:21:44 (20.7 MB/s) - rspamd’ saved [35180/35180]

That worked.
I was also going to create another website on mx1 (test.example.com), setup DNS, LE cert and see if I can access that externally.

Thanks for your help Till.