[SOLVED] Jobs in queue stuck - mariadb error reading communication packets

Discussion in 'ISPConfig 3 Priority Support' started by KoS, Oct 1, 2025.

  1. KoS

    KoS Member HowtoForge Supporter

    I first noticed that new jobs/changes on mail users are not being executed and stuck in the job queue.
    Than I saw in the logs of the cron job:
    Code:
    01.10.2025-11:05 - WARNING - There is already an instance of server.php running with pid 320507
    every minute. I manually stopped the job and started it from the commandline "php /usr/local/ispconfig/server/server.php" and it just "waits". When I cancel it by Ctrl+C I see in syslog the following error:
    Code:
    mariadbd[270067] 2025-10-01 13:08:23 1689 [Warning] Aborted connection 1689 to db: 'dbispconfig' user: 'ispconfig' host: 'localhost' (Got an error reading communication packets)
    I can see in the ispconfig cron.log that this issues started on Sep 28, but I don't see why. There have no package updates been done (dpkg.log). The server is a multi-server instance running Ubuntu 24.04 LTS (noble) and latest ISPconfig version 3.3.0p2.

    When I try with the mysql command on the command line with the same credentials as in the config file of ISPconfig I can normally access the database.

    I don't know where to best start debugging this issue, any hints would be great.
    Thanks!
     
  2. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    The read before posting article has instructions on how to debug stuck job queue.

     
  3. KoS

    KoS Member HowtoForge Supporter

  4. till

    till Super Moderator Staff Member ISPConfig Developer

    Please post the full debug output that you get when you run server.sh

    That's probably the result from canceling server.sh and not the cause of the issue, it just says you aborted the connection, which you did by pressing Ctrl+C.
     
  5. KoS

    KoS Member HowtoForge Supporter

    Thanks. But how do I get any debug output? As mentioned, when I run /usr/local/ispconfig/server/server.sh I don't get any output, it just "hangs" with empty output. Or the only output is the stale lockfile:
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

  7. KoS

    KoS Member HowtoForge Supporter

    Sorry I overlooked the debug sentence :-(
    The result/output is still the same: server.php hangs without any additional output.
    And in the job queue the change to debug level is still listed:
    upload_2025-10-2_11-47-33.png

    FYI: The setup is a multi-server setup, so the main server on which the ISPconfig UI is running is another one.
     
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    Please start server.sh and let it run. Maybe there is just a long-running job. You can also check in a second terminal, e.g., with the top command, what happens, e.g., how much load the server.sh is causing. If it causes load, then its doing something.
     
  9. KoS

    KoS Member HowtoForge Supporter

    The job ended after an hour or so, but still no output there and still the same jobs in the queue.
    I assume the "debug" setting is not being picked up so there is no additional output?
    There was not high load from server.php during the time it was running. And also no high load on the SQL server.
     
  10. till

    till Super Moderator Staff Member ISPConfig Developer

    Please verify that you can connect from the slave node to the master node using the 'mysql' command with the master server's hostname, username, password, and database name from the/usr/local/ispconfig/server/lib/config.inc.php file of the slave node.
     
  11. KoS

    KoS Member HowtoForge Supporter

    Yes, the connection to the master node works fine from that slave node with the credentials from the config file.
    Here the output of server.php with time measuring -> 121 minutes and "Unable to load the server configuration from database."?

    Code:
    time /usr/local/ispconfig/server/server.sh
    02.10.2025-16:53 - WARNING - There is already a lockfile set, but no process running with this pid (770805). Continuing.
    Unable to load the server configuration from database.
    real    121m41.175s
    user    0m0.027s
    sys     0m0.031s
    
     
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    Hard to say what the issue is. Seems as if the script has a database connection problem, try also if you can connect to the local mysql database on localhost and on 127.0.0.1 using the login details from config.inc.php file.
     
  13. KoS

    KoS Member HowtoForge Supporter

    I had already tested the local connection and it works fine (via localhost and 127.0.0.1).
    The other 2 slave nodes in the setup work fine, so I have no idea what is going wrong :-(
    Any way to debug the php script and see where it "hangs" for two hours?
     
  14. till

    till Super Moderator Staff Member ISPConfig Developer

    You couly try to add some lines like:

    Code:
    echo '1';
    
    .....
    
    echo '2';
    And so on in the server.php script to see where it hangs. Alternatively, you could contact Thom (@Th0m) from ISPConfig business support if you need further assistance with debugging this issue.
     
  15. KoS

    KoS Member HowtoForge Supporter

    Thanks @till
    The
    Code:
    Unable to load the server configuration from database.
    gave me the right position in the PHP file to debug and in the end it WAS an issue connecting to the master database. There was an MTU problem between the two hosts. Just connecting to the master server worked fine, but as soon as a query was executed it run into the MTU issue. Thanks to wireshark i found the root cause.
     
    ahrasis and till like this.
  16. till

    till Super Moderator Staff Member ISPConfig Developer

    Great to hear that you were able to solve it!
     

Share This Page