I state that the two servers that provide the email and web services do not have the ISPConfig web interface on themselves (Debian 12, Apache2, Postfix, Dovecot, ISPConfig 3.2.11) but are controlled by a system of three virtual machines that only provide the ISPConfig web interface service to precisely manage the servers with the services email and web. In the web interface of the panel I see the red dot that has been flashing for a few days with the number 3 inside and clicking on it says: Code: The following changes have not yet been pushed to the servers: Update server settings: 3 How do I figure out which changes have not yet been propagated (and I'm afraid they never will), which of the two servers they refer to and then remove the red dot? Thanks a lot
Enable debug mode on both servers, run server.sh on both servers, then you see which of the two systems is not fetching the changes from master. See: https://www.faqforge.com/linux/debugging-ispconfig-3-server-actions-in-case-of-a-failure/
After configuring the web panel for both servers, the red dot went from 3 to 7 and after a few minutes it became 5 and remained like this: I therefore deduce that the configuration has not been propagated to one of the two servers. This is the result of running a few minutes later (the red dot was already stable at 5) on each of the: ks06 Code: 17.08.2023-10:25 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'. 17.08.2023-10:25 - DEBUG [system.inc:2430] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/7.3/apache2/php.ini' - return code: 0 17.08.2023-10:25 - DEBUG [system.inc:2430] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/7.3/fpm/php.ini' - return code: 0 17.08.2023-10:25 - DEBUG [system.inc:2430] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/7.3/cgi/php.ini' - return code: 0 17.08.2023-10:25 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock finished server.php. ks07 Code: 17.08.2023-10:24 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'. 17.08.2023-10:24 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock finished server.php. Among other things, in the crontab of both servers the instruction is this: Code: * * * * * /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done and it's repeated identically twice on each server (of course I disabled them all). Can I delete one on each server or do I need both? Thanks a lot
According to the debug output, both servers are able to fetch changes. Do you have more than these two servers listed under System > server services? And the crontab is fine. and I doubt its identical. There must be two lines, one for server.sh and one for cron.sh.
In addition to the two service servers, the server that supplies the web interface of the panel is also displayed, i.e. the ISPConfig master, which is made up of 3 identical virtual machines (on three different servers, those of the services plus the backup one), with the database synchronized with galera cluster, which are seen by the system as a single server (in this way the panel is always usable and manages the virtual machine of the surviving services even in case of down of the other). No configuration has been done on this ISPConfig master server, it is never touched by anyone.
I also performed the configuration in the web panel of the third server: the red dot went from 5 to 7 and returned to 5 in a second. Here is the output of /usr/local/ispconfig/server/server.sh run on each of the three virtual machines that make up the third server (master ISPConfig): VM1 Code: root@ispc:~# /usr/local/ispconfig/server/server.sh 17.08.2023-11:12 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'. 17.08.2023-11:12 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock finished server.php. VM2 Code: root@ispc:~# /usr/local/ispconfig/server/server.sh 17.08.2023-11:12 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'. 17.08.2023-11:12 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock finished server.php. VM3 Code: root@ispc:~# /usr/local/ispconfig/server/server.sh 17.08.2023-11:12 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'. 17.08.2023-11:12 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock finished server.php. This structured system has been in production since 2018 and has never given any problems before upgrading to ISPConfig 3.2.11 and Debian 12.
The output is fine so far. Are these all systems connected via ISPConfig, or is there one additional server, as you wrote " (on three different servers, those of the services plus the backup one)" so if there is a fourth backup system which is e.g. turned off, then the pending changes might be for that system.
There is no other server besides these, either turned on or off. There are the two service servers that I posted earlier, ks06 and ks07 and then there is the ispc server that delivers the ISPConfig web panel which is made up of 3 virtual machines that are seen by the ISPConfig panel itself as a single server because the 3 virtual machines are identical (cloned, only the IP address changes) and the database is synchronized. I am attaching a screenshot of the panel
But each of the ispconfig servers has its own database, right? So you have a database dbispconfig, dbispconfig1, and dbispconfig2? It doe snot matter if these are on the same galera cluster or on the individual nodes, but there must be 3 databases as ISPConfig is not able to see which changes have been applied to which server otherwise and ISPConfig is taking care internally to move changes from master to the tow other databases itself.
You can get the result of the datalog status like this in master db: Code: SELECT COUNT( * ) AS cnt, sys_datalog.action, sys_datalog.dbtable FROM sys_datalog, server WHERE (server.server_id = sys_datalog.server_id or sys_datalog.server_id = 0) AND sys_datalog.user = 'admin' AND sys_datalog.datalog_id > server.updated GROUP BY sys_datalog.dbtable, sys_datalog.action in this example, it is queried for user admin.
In the ispc.webek.net server (which delivers the web panel) made up of the 3 ispc virtual machines, the ISPConfig database is identical and is called dbispconfig. In the servers of the services (web and email), ks06.webek.net and ks07.webek.net, the ISPConfig database is naturally different, respectively dbispconfigks06 and dbispconfigks07. So there are three physical servers, two dedicated to services and one dedicated to backups (ks08). An ispc virtual machine resides on each of the three physical servers and the ISPConfig system only sees ispc.webek.net. The virtual machines of the services, i.e. ks06.webek.net and ks07.webek.net, also reside on the other two physical servers. The dns configuration (managed manually for webek.net) ensures that each virtual machine of the services communicates with the ispc virtual machine which resides on its own physical server. So if the ks06 physical server fails, I can still access the ispc virtual machine that resides on the ks07 physical server and easily manage the virtual machine with the services that resides on the same server. Once the crashed server has been restored, any operations carried out on the ISPConfig web panel will automatically propagate to the restored services server (i.e. the ispc VM that had crashed will realign itself with the others thanks to database synchronization and the restored services server will go to take any operations to be performed). The third ispc VM, the one located on the physical backup server, serves only to make the MariaDB cluster odd (and as an additional backup copy). Here is the output of the mysql query executed on any of the ISPConfig master's VMs after typing USE dbispconfig; Code: +-----+--------+---------+ | cnt | action | dbtable | +-----+--------+---------+ | 5 | u | server | +-----+--------+---------+ 1 row in set (0,002 sec) I'm attaching the infrastructure schematic in case anyone else in the community needs it...
Please run this query so we can see which of the ndoes has unprocessed items: Code: SELECT sys_datalog.server_id, sys_datalog.datalog_id FROM sys_datalog, server WHERE (server.server_id = sys_datalog.server_id or sys_datalog.server_id = 0) AND sys_datalog.user = 'admin' AND sys_datalog.datalog_id > server.updated
Here is the output for each of the two service servers: ks06 Code: MariaDB [(none)]> USE dbispconfigks06; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed MariaDB [dbispconfigks06]> SELECT sys_datalog.server_id, sys_datalog.datalog_id FROM sys_datalog, server WHERE (server.server_id = sys_datalog.server_id or sys_datalog.server_id = 0) AND sys_datalog.user = 'admin' AND sys_datalog.datalog_id > server.updated; Empty set (0,002 sec) ks07 Code: MariaDB [(none)]> USE dbispconfigks07; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed MariaDB [dbispconfigks07]> SELECT sys_datalog.server_id, sys_datalog.datalog_id FROM sys_datalog, server WHERE (server.server_id = sys_datalog.server_id or sys_datalog.server_id = 0) AND sys_datalog.user = 'admin' AND sys_datalog.datalog_id > server.updated; Empty set (0,000 sec) But now I tried again to delete a test mailbox that I had created to see if everything worked: the changes were propagated to both servers of the services because on both disks in /var/vmail/ the mailbox was deleted and the red dot also disappeared as if the other pending operations had also been unblocked.
But I tried to bring the log level back to errors instead of debug and I was able to verify that ks07.webek.net is the server to which this type of operation is not propagated, in fact now there is again the red dot with the number 2. Reasoning on the sequence of events following Till's requests that followed this morning, probably the errors mode in the log levels was propagated to the ks07.webek.net server only after I deleted the test email box. I did various tests and in practice the server configurations changes to the ks07.webek.net server are propagated only after a change is made to the web and email services otherwise it remains there forever with the red dot flashing. Of course all this happens when the ispconfig instructions in the crontab are active otherwise the modification operations are not performed but this I think is normal. So there is something wrong in the configuration of ks07.webek.net probably inherent to its connection with the ISPConfig master because it seems to activate spontaneously only when the change involves other services that are not exclusively the server configuration.
As I mentioned above, ISPConfig nodes can not share a database, each of them must have a database that it is writing to exclusively. I refer here to the server part of ISPConfig, the interface part is different in the way that the interface is connected to the master only. With the diagram you showed, I'm not sure if really each server process has its exclusive own database where no other server process is writing to. If other processes are writing to the database or are sharing the server_id with another node, then all kind of strange side effects can appear as ISPConfig is unable to track changes then. Here is the process that an ISPConfig server node is doing: 1) Connect to master database and query it for all records from sys_datalog where server_id = its own server_id or server_id = 0 (which means the record has to be applied on all systems). 2) Now the node processes the changes and adds them to its own database the right sequence, that's why it is important each node has its own database. Otherwise it could happen that there is already data in the database which should not be there yet. 3) Finally, ISPConfig updates the column 'updated' in the 'server' table of the master database to the last datalog_id that it processed. So when you have pending records, then this means that the value in 'updated' column in 'server' table of master database is lower than the max. datalog_id from sys_datalog table (where server_id = 0 or server_id = the ID of this server). The likely reason why you have not noticed the problem yet is that ISPConfig versions < 3.2.11 did not took global changes into account when calculating pending actions, so in query from post #11, the part: (server.server_id = sys_datalog.server_id or sys_datalog.server_id = 0) was just: server.server_id = sys_datalog.server_id before. The reason for this change was that the red dot did not show pending records when you e.g. added a database user or a client. So as a temporary fix, you can alter SQL query in line 850 in the file /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php. Nut I think the change in the code we made is correct, so that's nothing that is likely to be changed back in ISPConfig code base as the underlying issue seems to be your custom database setup.
Thanks Till, I solved this way. I changed the dns pointing so that both virtual machines of the services always access a single master virtual machine which is also the only one that is used (again thanks to the dns) to manage the system from the web panel. In this way the whole system works as if there were two VMs for the services and only one master VM for the web interface. Of course, since the other two VMs are also running and always aligned, in the event of a down of the main server, i.e. the one on which both the main services VM and the master VM used reside, all the dns are moved to the secondary servers and therefore you will have the availability of the master VM on that server with full redundancy as well as the services also of the ISPConfig web interface while maintaining the system with the use of a single master VM at a time. This respects your assumption of a unique database for each ISPConfig or, in other words, every change that is made to the master ISPConfig database is managed by the ISPConfig installation itself. Thank you so much for your precious help and see you next time!
