Blocked jobqueue

The jobqueue keeps blocking as I'm putting my multi-server cluster together. I've tried the htf-common-issues script and it reports nothing untoward. I've tried running the server.sh script manually - with loglevel "debug" - and that's revealing nothing.

But then I tried "mysql -u ispcsrv2 -h node1.domain.tld" from node0 - I looked in the database with phpmyadmin to see that the user is "ispcsrv2" and that there is no password (field is blank). And I'm getting an "access denied".

Generally, MySQL / MariaDB as installed on Ubuntu 16.04 has been a pain because it locks down everything by default. As the multi-server install instructions suggest, I created "root" users for every IP address and hostname. I cleared the "plugin" that's installed that prevents remote logins. I also created a 'root'@'%' user so that you ought to be able to login as root from anywhere (with the correct password).

I can see, with phpMyAdmin, that the master server (node0) has created the "ispcsrv2" and "ispconfig" users. Node1 has an "ispconfig" user.

And, yes, I gave it over 15 minutes waiting for the job to disappear. It was definitely blocked. Indeed, I tried uninstalling and reinstalling (including manually clearing out the user entries with phpMyAdmin too) and it still happens.

I think what's happening is that the master server is getting an "access denied" to the other node, like I am when I try to login in manually from the command line, and this means the job blocks forever as it can never be completed.

Edit - Yes, when I run "uninstall.php" to try it again, then it says "access denied" with logging in as "ispcsrv2@node0" from node1. And the node doesn't disappear on node0's interface. So it's basically not getting through to the master server to let it know it's uninstalling.

 

I have run server.sh on the slave and it's repeatedly saying:
Access denied for user 'ipscsrv2'@'node1.domain.tld' (using password:YES).

But that does highlight something interesting. The user "ispcsrv2" is on node0, not node1. I'm accessing from node1, but the user is on node0. Is it trying to access "ispcsrv2" on its own database or something? And, if so, why?

 

The hosts file is correct and identical on both machines.

I'll reinstall again and make double sure that I've got the master hostname correct, I guess.

 

Okay, I've reinstalled and, on the slave, "server.sh" is saying:

PHP Warning: mysqli_connect(): (HY000/1045): Access denied for user 'ispcsrv2'@'slave.domain.tld' (using password: YES) in db_mysql.inc.php on line 79

And also the same error on line 85 five times in a row.

From the slave, I've tried "sudo mysql -u ipscsrv2 -h master.domain.tld" and it logs in just fine.

I notice that the error message says "using password: YES" but there is no password when I login. And I can see, via phpMyAdmin, that the password field is blank for both "ispcsrv2" entries (one the IP address, which is correct, and the other the hostname, which is correct).

The error messages suggest that the slave is trying to login to the slave, not the master. The host should be "master.domain.tld" - which I've manually tested works - but it says "slave.domain.tld" in the message.

 

$conf['dbmaster_host'] = 'master.domain.tld';

I do notice that it's got a random password listed ('dbmaster_password'), but the password field is blank for "ispcsrv2" when I look at it with phpMyAdmin.

 

Okay, I think I might just have solved it.

I noticed that none of these user accounts that ISPConfig was creating had passwords in phpMyAdmin - the password field was blank - but they are trying to set and use passwords, as I can see this with "server.sh" output.

So I checked the "plugin" field on the MySQL users and it's also blank. So I put "mysql_native_password" into the plugin field before adding the slave and now the user accounts have non-blank password fields and the job queue actually empties.

For some reason, on my set up, when new users are created, the "plugin" field is blank and not "mysql_native_password". But without "mysql_native_password", to explicitly tell it to use passwords for authorisation, it seems that MySQL won't let you set a password for a new user or login with the password properly.

A bit annoying that I've got to manually enter that into the database myself - I'll have to investigate if there's a way to set this globally somehow - but it does seem to work now that I've done that.