ISPConfig 3.1.15p2 is changing content of /etc/hosts and /etc/hostname on mirror node

Hi!
I'm currently having a strange issue with 2-node mirror setup with mysql bi-directional replication (on Ubuntu 18.04 and MySQL from MariaDB.ORG repo 10.3.18+maria~bionic).
The installation went pretty smooth according to the perfect guides for ubuntu and debian (8.4 cluster) and after some testing I even started migrating data over from my old server.
The problem occurs after trying to set up a relayhost on both nodes. On server1 the relayhost is properly set, but on server2 (which is mirror of server1) postconf -n |grep relayhost is empty and also no sasl_passwd* files are created...
So, I started a quick debug on server2 and this is the result:

On Server2 after setting a relayhost in controlpanel (Server Config-> server2.example.tld -> Mail) the following was logged (anonymized + shortened):

Code:

root@server2:~# /usr/local/ispconfig/server/server.sh
02.11.2019-14:27 - WARNING - There is already a lockfile set, but no process running with this pid (3876). Continuing.
[...]
02.11.2019-14:27 - DEBUG - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
02.11.2019-14:27 - DEBUG - Network configuration disabled in server settings.
02.11.2019-14:27 - DEBUG - Changed /etc/hostname to server2.example.tld
02.11.2019-14:27 - DEBUG - Changed /etc/mailname to server2.example.tld
02.11.2019-14:27 - DEBUG - safe_exec cmd: postconf -e 'myhostname = server2.example.tld' - return code: 0
02.11.2019-14:27 - DEBUG - safe_exec cmd: postconf -e 'mydestination = server2.example.tld, localhost, localhost.localdomain' - return code: 0
02.11.2019-14:27 - DEBUG - Changed changed myhostname and mydestination in postfix main.cf to server2.example.tld
02.11.2019-14:27 - DEBUG - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'.
02.11.2019-14:27 - DEBUG - safe_exec cmd: which 'dovecot' 2> /dev/null - return code: 0
02.11.2019-14:27 - DEBUG - Calling function 'update' from plugin 'xmpp_plugin' raised by event 'server_update'.
02.11.2019-14:27 - DEBUG - Processed datalog_id 669
[...]
02.11.2019-14:27 - DEBUG - Processed datalog_id 671
[...]
02.11.2019-14:27 - DEBUG - safe_exec cmd: postconf -e 'smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination, reject_rbl_client zen.spamhaus.org, check_recipient_access mysql:/etc/postfix/mysql-virtual_recipient.cf, check_recipient_access mysql:/etc/postfix/mysql-virtual_policy_greylist.cf' - return code: 0
postfix/postfix-script: refreshing the Postfix mail system
[...]
02.11.2019-14:27 - DEBUG - Processed datalog_id 673
02.11.2019-14:27 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
02.11.2019-14:27 - DEBUG - Restarting httpd: systemctl restart apache2.service
02.11.2019-14:27 - DEBUG - Calling function 'restartPostfix' from module 'mail_module'.
[...]
02.11.2019-14:27 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
finished.
PHP Warning:  mysqli_real_connect(): (HY000/2002): Connection timed out in /usr/local/ispconfig/server/lib/classes/db_mysql.inc.php on line 85
PHP Warning:  mysqli_real_connect(): (HY000/2002): Connection timed out in /usr/local/ispconfig/server/lib/classes/db_mysql.inc.php on line 91
[...]
Database connection failedPHP Warning:  mysqli_real_connect(): (HY000/2002): Connection timed out in /usr/local/ispconfig/server/lib/classes/db_mysql.inc.php on line 91
Database connection failedZugriff auf Datenbankserver fehlgeschlagen! / Database server not accessible!

After this the files /etc/hosts and /etc/hostname changed like so:

Code:

root@server2:~# cat /etc/hosts
127.0.0.1 	localhost
10.0.0.10 	server1.example.tld server2 	<-- Changed from: 10.0.0.10 	server2.example.tld server2
192.168.11.217	server1.example.tld server1
[...]

root@server2:~# cat /etc/hostname
server2.example.tld				<-- Changed from: server2

I presume Database connection failed message is due to the wrong hostnames in /etc/hosts and /etc/hostname.

So, I changed back the files to their original values, protected them with

And debugged again:

Code:

root@server2:~# /usr/local/ispconfig/server/server.sh
02.11.2019-16:23 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
02.11.2019-16:23 - DEBUG - Found 2 changes, starting update process.
02.11.2019-16:23 - DEBUG - Replicated from master: REPLACE INTO `server` (`server_id`,[...])
02.11.2019-16:23 - DEBUG - Calling function 'server_ip' from plugin 'apache2_plugin' raised by event 'server_update'.
02.11.2019-16:23 - DEBUG - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0
02.11.2019-16:23 - DEBUG - Writing the conf file: /etc/apache2/sites-available/ispconfig.conf
02.11.2019-16:23 - DEBUG - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
02.11.2019-16:23 - DEBUG - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0
02.11.2019-16:23 - DEBUG - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
02.11.2019-16:23 - DEBUG - Network configuration disabled in server settings.
PHP Warning:  unlink(/etc/hostname): Operation not permitted in /usr/local/ispconfig/server/lib/classes/system.inc.php on line 864
PHP Warning:  file_put_contents(/etc/hostname): failed to open stream: Operation not permitted in /usr/local/ispconfig/server/lib/classes/system.inc.php on line 865
02.11.2019-16:23 - DEBUG - Changed /etc/hostname to server2.example.tld
02.11.2019-16:23 - DEBUG - Changed /etc/mailname to server2.example.tld
02.11.2019-16:23 - DEBUG - safe_exec cmd: postconf -e 'myhostname = server2.example.tld' - return code: 0
02.11.2019-16:23 - DEBUG - safe_exec cmd: postconf -e 'mydestination = server2.example.tld, localhost, localhost.localdomain' - return code: 0
02.11.2019-16:23 - DEBUG - Changed changed myhostname and mydestination in postfix main.cf to server2.example.tld
PHP Warning:  unlink(/etc/hosts): Operation not permitted in /usr/local/ispconfig/server/lib/classes/system.inc.php on line 864
PHP Warning:  file_put_contents(/etc/hosts): failed to open stream: Operation not permitted in /usr/local/ispconfig/server/lib/classes/system.inc.php on line 865
02.11.2019-16:23 - DEBUG - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'.
02.11.2019-16:23 - DEBUG - safe_exec cmd: postconf -e 'relayhost = ' - return code: 0
02.11.2019-16:23 - DEBUG - safe_exec cmd: which 'dovecot' 2> /dev/null - return code: 0
02.11.2019-16:23 - DEBUG - Calling function 'update' from plugin 'xmpp_plugin' raised by event 'server_update'.
02.11.2019-16:23 - DEBUG - Processed datalog_id 683
02.11.2019-16:23 - DEBUG - Replicated from master: REPLACE INTO `server` (`server_id`,[...])
02.11.2019-16:23 - DEBUG - Calling function 'server_ip' from plugin 'apache2_plugin' raised by event 'server_update'.
02.11.2019-16:23 - DEBUG - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0
02.11.2019-16:23 - DEBUG - Writing the conf file: /etc/apache2/sites-available/ispconfig.conf
02.11.2019-16:23 - DEBUG - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
02.11.2019-16:23 - DEBUG - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0
02.11.2019-16:23 - DEBUG - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
02.11.2019-16:23 - DEBUG - Network configuration disabled in server settings.
PHP Warning:  unlink(/etc/hostname): Operation not permitted in /usr/local/ispconfig/server/lib/classes/system.inc.php on line 864
PHP Warning:  file_put_contents(/etc/hostname): failed to open stream: Operation not permitted in /usr/local/ispconfig/server/lib/classes/system.inc.php on line 865
02.11.2019-16:23 - DEBUG - Changed /etc/hostname to server2.example.tld
02.11.2019-16:23 - DEBUG - Changed /etc/mailname to server2.example.tld
02.11.2019-16:23 - DEBUG - safe_exec cmd: postconf -e 'myhostname = server2.example.tld' - return code: 0
02.11.2019-16:23 - DEBUG - safe_exec cmd: postconf -e 'mydestination = server2.example.tld, localhost, localhost.localdomain' - return code: 0
02.11.2019-16:23 - DEBUG - Changed changed myhostname and mydestination in postfix main.cf to server2.example.tld
PHP Warning:  unlink(/etc/hosts): Operation not permitted in /usr/local/ispconfig/server/lib/classes/system.inc.php on line 864
PHP Warning:  file_put_contents(/etc/hosts): failed to open stream: Operation not permitted in /usr/local/ispconfig/server/lib/classes/system.inc.php on line 865
02.11.2019-16:23 - DEBUG - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'.
02.11.2019-16:23 - DEBUG - safe_exec cmd: postconf -e 'relayhost = ' - return code: 0
02.11.2019-16:23 - DEBUG - safe_exec cmd: which 'dovecot' 2> /dev/null - return code: 0
02.11.2019-16:23 - DEBUG - Calling function 'update' from plugin 'xmpp_plugin' raised by event 'server_update'.
02.11.2019-16:23 - DEBUG - Processed datalog_id 685
02.11.2019-16:23 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
02.11.2019-16:23 - DEBUG - Restarting httpd: systemctl restart apache2.service
02.11.2019-16:23 - DEBUG - Calling function 'restartPostfix' from module 'mail_module'.
[...]
02.11.2019-16:23 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock

For some reason the php function file_put_contents is trying to modify /etc/hosts and /etc/hostname on the mirror node and if not protected it succeeds...

I do not have much experience with ISPConfig setups. Should this happen if the "network configuration option" is NOT set?
I'd appreciate some thoughts and points into the right direction.

Thanks!

Regards,
BG

 

Similar problem here...
My mirror node has also stopped responding to pings/ssh/anything, once I restarted and checked the /etc/hosts file I noticed that the line corresponding to the "master" node was modified: it reported the mirror node hostname itself!

 

My workaround was to remove on both nodes: