"The following changes are not yet populated to all servers:" not going away

Hello,
After making some changes to some sites, the "The following changes are not yet populated to all servers:" blinking icon will not go away. The Job Queue is empty. The changes I made appear to be working, so I think it is just a matter of the "alert" not being able to be clear for some reason.

How do I make this go away?

Thanks!

 

It is a multi-node system, consisting of just two nodes: a control panel server and a web server (hosting MySQL databases as well).

Both nodes are online and reachable.

Clicking in the alert shows me:

datalog_status_i_client: 1
datalog_status_u_client: 1
datalog_status_i_sys_group: 1

 

Debug level logging was already enabled. No errors are showing up in the System-Log

Here is what running server.sh manually on the web server node outputs:

15.10.2023-10:53 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
15.10.2023-10:53 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
finished server.php.

(which is the same output I see in "System-Log" in the Control Panel)

There don't *appear* to be any errors...?

 

OK, that was a bit more enlightening. Here is the running of '/usr/local/ispconfig/server/server.sh' on the control panel server::

15.10.2023-14:29 - WARNING - There is already a lockfile set, but no process running with this pid (17999). Continuing.
Database connection failed Database connection failed Database connection failed Database connection failed Database connection failed Zugriff auf Datenbankserver fehlgeschlagen! / Database server not accessible! PHP Fatal error: Uncaught Error: Call to a member function testConnection() on bool in /usr/local/ispconfig/server/server.php:64
Stack trace:
#0 {main}
thrown in /usr/local/ispconfig/server/server.php on line 64
Database connection failed Database connection failed Database connection failed Database connection failed Database connection failed Zugriff auf Datenbankserver fehlgeschlagen! / Database server not accessible!

So something is wrong with the connection to the database. What is the best way to go about resolving that?

 

I am able to log in MySQL on the control panel server with the info in the '/usr/local/ispconfig/interface/lib/config.inc.php' file, so it does not appear to be an issue with the database credentials.

I can log in via both 'localhost' and '127.0.0.1'

 

I've had the same error recently.

I assume server.php is using a hostname for the sql server to connect to?

That hostname isn't resolving to the ip that you are expecting, hence there's no user with the host/ip in sql that server.php is expecting and therefor can't connect.

 

OK, I was not able to connect via the MySQL client from the slave (web server node) to the master (control panel node) with the "master database" credentials in '/usr/local/ispconfig/server/lib/config.inc.php' (from the slave node). I ran the "GRANT ALL PRIVILEGES" command on the master node's MySQL to allow connections from the slave's IP, and I can now connect with the MySQL client from the slave to the master with the "master database" credentials.

Hrm, some helpful information might be that I recently changed the IP addresses of the master and slave nodes, as we moved them (VMs) to a different datacenter. Sorry, should have mentioned that. When I changed the IPs, I followed the steps from another post and updated the info in the Control Panel GUI and /etc/hosts

Running '/usr/local/ispconfig/server/server.sh' on the master node still gives an error being able to connect to the database:

--- START ---
Database connection failed Database connection failed Database connection failed Database connection failed Database connection failed Zugriff auf Datenbankserver fehlgeschlagen! / Database server not accessible! PHP Fatal error: Uncaught Error: Call to a member function testConnection() on bool in /usr/local/ispconfig/server/server.php:64
Stack trace:
#0 {main}
thrown in /usr/local/ispconfig/server/server.php on line 64
Database connection failed Database connection failed Database connection failed
--- END ---

Here is line 64+ from '/usr/local/ispconfig/server/server.php':

--- START ---
if ($app->dbmaster->testConnection()) {
$server_db_record = $app->dbmaster->queryOneRecord("SELECT * FROM server WHERE server_id = ?", $conf['server_id']);

if(!is_array($server_db_record)) die('Unable to load the server configuration from database.');
--- END ---

 

OK, so just found something interesting: the '/usr/local/ispconfig/server/lib/config.inc.php' have different info for the "master DB"

On the master node, 'dbmaster_port' and 'dbmaster_user' are blank, whereas those values are set in the slave node's file. But more interestingly, the dbmaster_password' value on both nodes are different. I suspect I need to change one to the other, but am not sure which? My gut tells me I should update the slave node to match the value on the master node?

 

OK, so that was a red herring. I checked the 'config.inc.php' files in question on the original VMs in the "old" datacenter, and the different passwords I am currently seeing are on the originals as well.

 

Is there something in the 'dbispconfig' database itself that allows connections?

I did some testing on the original VMs, and I am able to connect with the MySQL client to the master DB from the slave node with the credentials in '/usr/local/ispconfig/server/lib/config.inc.php' without having to have done anyting on the master node's MySQL.

If there is something in the 'dbispconfig' database itself that is allowing the access, which table is it? I can manually update the value.

 

Those are indeed empty pn the master node.

 

Yes. When I run '/usr/local/ispconfig/server/server.sh' manually on the master, I get the database connection issue. When I run that same file on the slave, there is no error and ends with "finished server.php."

On the master, I can connect to the dbispconfig database on it with the credentials in '/usr/local/ispconfig/server/lib/config.inc.php'. I am also able to connect to the dbispconfig database on the slave from the master using the credentials in '/usr/local/ispconfig/server/lib/config.inc.php' on the slave.

I am puzzled as to what database connection it is unable to make.

 

So using the MySQL client, if I am able to connect to the dbispconfig database on the master node FROM the master node using the credentials in '/usr/local/ispconfig/server/lib/config.php.inc' (on the master node), what would cause the error of not being able to connect to the database when running the '/usr/local/ispconfig/server/server.sh' file (though technically it is the running of '/usr/local/ispconfig/server/server.php' that is running into the error)