[SOLVED]Fix job queue stuck

Discussion in 'General' started by Jean-François Questiaux, Jul 11, 2024.

  1. Hi,
    I have a multiserver setup but lately the job queue is always stuck and I can't change anything.
    Running "/usr/local/ispconfig/server/server.sh" on the server where the update should be performed, I got this error:
    Database connection failed Database connection failed Database connection failed Database connection failed Database connection failed Zugriff auf Datenbankserver fehlgeschlagen! / Database server not accessible! PHP Fatal error: Uncaught Error: Call to a member function testConnection() on bool in /usr/local/ispconfig/server/server.php:67
    Stack trace:
    #0 {main}
    thrown in /usr/local/ispconfig/server/server.php on line 67​

    So I guess it's a permission error but I have no idea how to fix this.
    i'm running ISPConfig 3.2.12 on all servers. Servers run with Debian 10/11/12 (Master is Debian 11, the exemple above is also Debian 11).
    Thanks in advance for your help.
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    Have you checked if mysql or mariadb is started?
     
  3. Yes, MariaDb is running on all servers.
     
  4. Taleman

    Taleman Well-Known Member HowtoForge Supporter

  5. When I try to force the update of ISPConfig on the "slave" server, I get "Unable to connect to mysql server Connection refused".
    However, the credentials are correct since I can access the server with an external tool (like Sequel Pro for instance).
     
    Last edited: Jul 11, 2024
  6. I tried another change on one of the other "slave" server (and it got stuck again), then I ran the report:
    Code:
    ##### SERVER #####
    IP-address (as per hostname): ***.***.***.***
    [WARN] could not determine server's ip address by ifconfig
    [INFO] OS version is Debian GNU/Linux 12 (bookworm)
     
    [INFO] uptime:  13:59:43 up 17:15,  0 user,  load average: 0.26, 0.19, 0.09
     
    [INFO] memory:
                   total        used        free      shared  buff/cache   available
    Mem:           3.9Gi       620Mi       1.1Gi        65Mi       2.3Gi       3.3Gi
    Swap:          1.0Gi          0B       1.0Gi
     
    [INFO] systemd failed services status:
      UNIT                  LOAD   ACTIVE SUB    DESCRIPTION
    ● console-setup.service loaded failed failed Set console font and keymap
    
    LOAD   = Reflects whether the unit definition was properly loaded.
    ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
    SUB    = The low-level unit activation state, values depend on unit type.
    1 loaded units listed.
    
    [INFO] ISPConfig is installed.
    
    ##### ISPCONFIG #####
    ISPConfig version is 3.2.12
    
    
    ##### VERSION CHECK #####
    
    [INFO] php (cli) version is 8.2.21
    [INFO] php-cgi (used for cgi php in default vhost!) is version 8.2.21
    
    ##### PORT CHECK #####
    
    [WARN] Port 8080 (ISPConfig) seems NOT to be listening
    [WARN] Port 143 (IMAP server) seems NOT to be listening
    [WARN] Port 993 (IMAP server SSL) seems NOT to be listening
    [WARN] Port 110 (POP3 server) seems NOT to be listening
    [WARN] Port 995 (POP3 server SSL) seems NOT to be listening
    [WARN] Port 465 (SMTP server SSL) seems NOT to be listening
    [WARN] Port 21 (FTP server) seems NOT to be listening
    
    ##### MAIL SERVER CHECK #####
    
    [WARN] I found no "smtp" entry in your postfix master.cf
    [WARN] I found no "submission" entry in your postfix master.cf
    [INFO] this is not critical, but if you want to offer port 587 for smtp connections you have to enable this.
    [WARN] I found no "smtps" entry in your postfix master.cf
    [INFO] this is not critical, but if you want to offer SSL for smtp (not TLS) connections you have to enable this.
    
    ##### RUNNING SERVER PROCESSES #####
    
    [INFO] I found the following web server(s):
            Apache 2 (PID 966)
    [INFO] I found the following mail server(s):
            Exim (PID 185338)
    [WARN] I could not determine which pop3 server is running.
    [WARN] I could not determine which imap server is running.
    [WARN] I could not determine which ftp server is running.
    
    ##### LISTENING PORTS #####
    (only           ()
    Local           (Address)
    [localhost]:3306                (752/mariadbd)
    [anywhere]:10000                (1382/perl)
    [localhost]:25          (185338/exim4)
    [anywhere]:22           (651/sshd:)
    [anywhere]:556          (194923/perl)
    [anywhere]:555          (192200/perl)
    [localhost]:11211               (611/memcached)
    *:*:*:*::*:25           (185338/exim4)
    *:*:*:*::*:8081         (966/apache2)
    [localhost]0000         (1382/perl)
    *:*:*:*::*:80           (966/apache2)
    *:*:*:*::*:22           (651/sshd:)
    *:*:*:*::*:443          (966/apache2)
    
    
    
    
    ##### IPTABLES #####
    Chain INPUT (policy ACCEPT)
    target     prot opt source               destination         
    f2b-sshd   6    --  [anywhere]/0            [anywhere]/0            multiport dports 22
    
    Chain FORWARD (policy ACCEPT)
    target     prot opt source               destination         
    
    Chain OUTPUT (policy ACCEPT)
    target     prot opt source               destination         
    
    Chain f2b-sshd (1 references)
    target     prot opt source               destination         
    RETURN     0    --  [anywhere]/0            [anywhere]/0           
    
    
    
    
    ##### LET'S ENCRYPT #####
    acme.sh is installed in /root/.acme.sh/acme.sh
    What I find strange is that it does not detect the MariaDB server in the ***SERVER*** section. And this server IS running (this server is a web and db server).
     
  7. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    It is not reported there, that is normal.
    Db server is running, see later under Listening ports.
    Code:
    [localhost]:3306                (752/mariadbd)
    Did you run common issues script also on all your ISPConfig servers?
    You wrote:
    Maybe those credentials you used are correct, but ISPConfig uses wrong credentials. Have you changed the database password sometime after installing ISPConfig?
    Did you check database service is running on all servers? For example with
    Code:
    systemctl status mariadb.service
    Check the database setting ISPConfig uses are such that they do work when connecting, they are in file /usr/local/ispconfig/server/lib/mysql_clientdb.conf .
     
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    Most likely, the password of the MySQL user "ispconfig" has been changed; you can find it in the file /usr/local/ispconfig/server/lib/config.inc.php
     
  9. On the "master" server, the password in the mysql table is still the same as the one in the /usr/local/ispconfig/server/lib/config.inc.php file.
    What I don't understand is that yesterday, I was able to update all these server from ISPConfig 3.2.11p to 3.2.12 and today, using the same procedure the connection is refused.

    I have also updated the master server from Debian 10 > 11 (following the procedure described here: https://www.howtoforge.com/how-to-upgrade-from-debian-10-to-debian-11/). Could it be the problem?
     
  10. till

    till Super Moderator Staff Member ISPConfig Developer

    This error is not about the master server, its an error on the affected slave system. This slave system cannot connect to its local mysql database anymore. Check the config.inc.php file I mentioned and test the mysql login using the mysql command.
     
  11. So, on the 5 servers (1 master and 4 slaves), I did "mysql -u ispconfig -p", with the password I found in /usr/local/ispconfig/server/lib/config.inc.php and every time I logged in correctly.
     
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    You must use the exact settings from that file, means to also specify the hostname of the database. Besides that, testing it on all systems is not needed and does not help, just test it on the system that fails.

    If this works with the exact details from that file, try the details from dbmaster section of that file to see if you can connect to the master server.
     
  13. OK, that's interesting because, if I enter this (XXXXX is the actual and correct password of course):
    mysql -h localhost -u ispconfig -ppassword XXXXX --port=3306​
    I get this error:
    ERROR 1045 (28000): Access denied for user 'ispconfig'@'localhost' (using password: YES)​
    while if I enter this:
    mysql -h localhost -u ispconfig -p --port=3306​
    then enter the password at the prompt, the login goes through fine.

    And I tested on all the servers because they all fail.
     
  14. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

  15. My mistake! There is an error in my last post. The correct command is of course:
    mysql -h localhost -u ispconfig -pXXXX --port=3306​
    and it's working fine.
    What I don't understand is that whole system has worked for years and now it's not working at all anymore. I have a feeling that it happened when I updated form 3.2.11p to 3.2.12. One main change I see is that the "user" table in mysql database changed from a "normal" table to a view, except on one of the server (I guess it's related to the MariaDB version). Could it be the problem?

    For info, here are the versions of the servers:
    Master : Debian 11 - MariaDB 10.5.23 - ISPConfig 3.2.12
    Slaves:
    (mail server) Debian 11 - MariaDB 10.5.23 - ISPConfig 3.2.12
    (db server) Debian 10 - MariaDB 10.3.39 - ISPConfig 3.2.12
    (web server) Debian 11 - MariaDB 10.5.23 - ISPConfig 3.2.12
    (web & db server) Debian 12 - MariaDB 10.11.6 - ISPConfig 3.2.11p (I restored the backup before the update to see if that helps, but no and I can't update anymore, the ispconfig_update.sh fails "Unable to connect to mysql server Connection refused")​
     
  16. till

    till Super Moderator Staff Member ISPConfig Developer

    Did you test the connection to the master now as I suggested in post #12?
     
  17. Sorry I overlooked that part.
    And yes, this is where the problem is :
    mysql -h serv13master.xxxx.fr -u ispcsrv3 -pXXXXXX --port=3306​
    returns:
    ERROR 2002 (HY000): Can't connect to MySQL server on 'serv13master.xxxx.fr' (115)​

    Yet the ISPConsole on this server is accessible, as well as Roundcube.
     
  18. pyte

    pyte Well-Known Member HowtoForge Supporter

    Does that server allow connecting on port 3306 from the system you are trying to connect from? Did you check your firewall settings? You need to make sure that the firewall ports are open on the machine and any firewall that may be in between the two systems
     
  19. till

    till Super Moderator Staff Member ISPConfig Developer

    Check that networking is not disabled in mysql on the master (skip-networking option not set) and that mysql listens on all interfaces on the master and that you have not closed the mysql port in a firewall on the master.
     
  20. Well, thank you so much!
    skip-networking was OFF but bind_address was set on 127.0.0.0! It seems that it is the new default parameter for MariaDB. It must have changed when I updated the server from Debian 10 to 11.
    Anyway, now mysql -h serv13master.xxxx.fr -u ispcsrv3 -pXXXXXX --port=3306 is working, so I guess the rest also. I'll try that.

    Thanks again.
     
    pyte and till like this.

Share This Page