Job Queue stalled

Discussion in 'Installation/Configuration' started by nzAndy, Sep 23, 2023.

  1. nzAndy

    nzAndy Member

    I have a job queue backlog that is not clearing in ISPConfig 3, its been like this for a couple of hours.
    I went part way through the debugging of server actions tutorial (should I be following this to fix the issue?), can anybody see anything below that could be causing the stalled job queue issue, thanks


    -----------------------------------------

    Sep 23 07:00:03 yakitori systemd[1]: apt-news.service: Deactivated successfully.
    Sep 23 07:00:03 yakitori systemd[1]: Finished Update APT News.
    Sep 23 07:00:05 yakitori systemd[1]: esm-cache.service: Deactivated successfully.
    Sep 23 07:00:05 yakitori systemd[1]: Finished Update the local ESM caches.
    Sep 23 07:00:08 yakitori systemd-resolved[826]: Grace period over, resuming full feature set (UDP+EDNS0) for DNS server 127.0.0.1.
    Sep 23 07:00:12 yakitori postfix/smtpd[7831]: connect from unknown[94.156.102.80]
    Sep 23 07:00:15 yakitori named[1060]: shut down hung fetch while resolving '_https._tcp.deb.goaccess.io/SRV'
    Sep 23 07:00:16 yakitori postfix/smtpd[7831]: warning: unknown[94.156.102.80]: SASL LOGIN authentication failed: UGFzc3dvcmQ6
    Sep 23 07:00:16 yakitori postfix/smtpd[7831]: disconnect from unknown[94.156.102.80] ehlo=1 auth=0/1 quit=1 commands=2/3
    Sep 23 07:00:41 yakitori kernel: [ 4604.797752] [UFW BLOCK] IN=eno1 OUT= MAC=01:00:5e:00:00:01:bc:30:d9:a7:dd:da:08:00 SRC=192.168.0.1 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0x00 TTL=1 ID=38449 DF PROTO=2
    Sep 23 07:01:01 yakitori CRON[8603]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:01:01 yakitori CRON[8604]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:02:01 yakitori CRON[8633]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:02:01 yakitori CRON[8634]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:02:03 yakitori postfix/smtpd[8660]: connect from unknown[94.156.102.78]
    Sep 23 07:02:07 yakitori postfix/smtpd[8660]: warning: unknown[94.156.102.78]: SASL LOGIN authentication failed: UGFzc3dvcmQ6
    Sep 23 07:02:07 yakitori postfix/smtpd[8660]: disconnect from unknown[94.156.102.78] ehlo=1 auth=0/1 quit=1 commands=2/3
    Sep 23 07:02:46 yakitori kernel: [ 4729.798935] [UFW BLOCK] IN=eno1 OUT= MAC=01:00:5e:00:00:01:bc:30:d9:a7:dd:da:08:00 SRC=192.168.0.1 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0x00 TTL=1 ID=17876 DF PROTO=2
    Sep 23 07:03:01 yakitori CRON[8665]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:03:01 yakitori CRON[8666]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:04:01 yakitori CRON[8696]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:04:01 yakitori CRON[8695]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:04:51 yakitori kernel: [ 4854.800079] [UFW BLOCK] IN=eno1 OUT= MAC=01:00:5e:00:00:01:bc:30:d9:a7:dd:da:08:00 SRC=192.168.0.1 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0x00 TTL=1 ID=63874 DF PROTO=2
    Sep 23 07:05:01 yakitori CRON[8730]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:05:01 yakitori CRON[8736]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:05:01 yakitori CRON[8737]: (getmail) CMD (/usr/local/bin/run-getmail.sh > /dev/null 2>> /dev/null)
    Sep 23 07:05:02 yakitori pure-ftpd: ([email protected]) [INFO] New connection from 127.0.0.1
    Sep 23 07:05:02 yakitori pure-ftpd: ([email protected]) [INFO] Logout.
    Sep 23 07:05:02 yakitori dovecot: imap-login: Disconnected: Connection closed (disconnected before auth was ready, waited 0 secs): user=<>, rip=127.0.0.1, lip=127.0.0.1, secured, session=<28MTWAEGwr5/AAAB>
    Sep 23 07:05:02 yakitori dovecot: pop3-login: Disconnected: Connection closed (no auth attempts in 0 secs): user=<>, rip=127.0.0.1, lip=127.0.0.1, secured, session=<YOYTWAEGsp5/AAAB>
    Sep 23 07:05:02 yakitori postfix/smtpd[8840]: connect from localhost.localdomain[127.0.0.1]
    Sep 23 07:05:02 yakitori postfix/smtpd[8840]: lost connection after CONNECT from localhost.localdomain[127.0.0.1]
    Sep 23 07:05:02 yakitori postfix/smtpd[8840]: disconnect from localhost.localdomain[127.0.0.1] commands=0/0
    Sep 23 07:05:27 yakitori postfix/anvil[7085]: statistics: max connection rate 1/60s for (smtp:51.89.157.6) at Sep 23 06:58:08
    Sep 23 07:05:27 yakitori postfix/anvil[7085]: statistics: max connection count 1 for (smtp:51.89.157.6) at Sep 23 06:58:08
    Sep 23 07:05:27 yakitori postfix/anvil[7085]: statistics: max cache size 1 at Sep 23 06:58:08
    Sep 23 07:06:01 yakitori CRON[8849]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:06:01 yakitori CRON[8848]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:06:56 yakitori kernel: [ 4979.801234] [UFW BLOCK] IN=eno1 OUT= MAC=01:00:5e:00:00:01:bc:30:d9:a7:dd:da:08:00 SRC=192.168.0.1 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0x00 TTL=1 ID=49440 DF PROTO=2
    Sep 23 07:07:01 yakitori CRON[8901]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:07:01 yakitori CRON[8902]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:08:01 yakitori CRON[9098]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:08:01 yakitori CRON[9099]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:09:00 yakitori systemd[1]: Starting Clean php session files...
    Sep 23 07:09:01 yakitori kernel: [ 5104.802551] [UFW BLOCK] IN=eno1 OUT= MAC=01:00:5e:00:00:01:bc:30:d9:a7:dd:da:08:00 SRC=192.168.0.1 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0x00 TTL=1 ID=31910 DF PROTO=2
    Sep 23 07:09:01 yakitori systemd[1]: phpsessionclean.service: Deactivated successfully.
    Sep 23 07:09:01 yakitori systemd[1]: Finished Clean php session files.
    Sep 23 07:09:01 yakitori systemd[1]: phpsessionclean.service: Consumed 1.145s CPU time.
    Sep 23 07:09:01 yakitori CRON[9670]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:09:01 yakitori CRON[9669]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
    Sep 23 07:09:01 yakitori CRON[9671]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:09:10 yakitori postfix/smtpd[9747]: connect from unknown[193.42.33.87]
    Sep 23 07:09:13 yakitori postfix/smtpd[9747]: warning: unknown[193.42.33.87]: SASL LOGIN authentication failed: UGFzc3dvcmQ6
    Sep 23 07:09:13 yakitori postfix/smtpd[9747]: disconnect from unknown[193.42.33.87] ehlo=1 auth=0/1 quit=1 commands=2/3
    Sep 23 07:09:15 yakitori postfix/smtpd[9747]: connect from ip6.ip-51-89-157.eu[51.89.157.6]
    Sep 23 07:09:18 yakitori postfix/smtpd[9747]: warning: ip6.ip-51-89-157.eu[51.89.157.6]: SASL LOGIN authentication failed: UGFzc3dvcmQ6
    Sep 23 07:09:18 yakitori postfix/smtpd[9747]: disconnect from ip6.ip-51-89-157.eu[51.89.157.6] ehlo=1 auth=0/1 quit=1 commands=2/3
    Sep 23 07:09:58 yakitori postfix/qmgr[1935]: 3137280F60: from=<[email protected]>, size=2271, nrcpt=1 (queue active)
    Sep 23 07:09:58 yakitori postfix/smtp[9803]: 3137280F60: host extmail.bigpond.com[203.42.40.138] refused to talk to me: 550-5.7.1 Connection refused - IB119. 58.169.29.57 is blacklisted in Abusix 550-5.7.1 Mail Intelligence. See https://lookup.abusix.com/search?q=58.169.29.57 550 5.7.1 i{0b3a663d-1c86-4efc-bf10-262ac8e8e544}
    Sep 23 07:09:59 yakitori postfix/smtp[9803]: 3137280F60: to=<[email protected]>, relay=extmail.bigpond.com[203.42.22.10]:25, delay=319794, delays=319793/0.03/0.48/0, dsn=4.7.1, status=deferred (host extmail.bigpond.com[203.42.22.10] refused to talk to me: 550-5.7.1 Connection refused - IB119. 58.169.29.57 is blacklisted in Abusix 550-5.7.1 Mail Intelligence. See https://lookup.abusix.com/search?q=58.169.29.57 550 5.7.1 i{4a9afa15-e33b-40ee-97d1-b26432d58d47})
    Sep 23 07:10:01 yakitori CRON[9839]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:10:01 yakitori CRON[9838]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:10:01 yakitori CRON[9840]: (getmail) CMD (/usr/local/bin/run-getmail.sh > /dev/null 2>> /dev/null)
    Sep 23 07:10:02 yakitori pure-ftpd: ([email protected]) [INFO] New connection from 127.0.0.1
    Sep 23 07:10:02 yakitori pure-ftpd: ([email protected]) [INFO] Logout.
    Sep 23 07:10:02 yakitori postfix/smtpd[9747]: connect from localhost.localdomain[127.0.0.1]
    Sep 23 07:10:02 yakitori postfix/smtpd[9747]: lost connection after CONNECT from localhost.localdomain[127.0.0.1]
    Sep 23 07:10:02 yakitori postfix/smtpd[9747]: disconnect from localhost.localdomain[127.0.0.1] commands=0/0
    Sep 23 07:10:03 yakitori dovecot: pop3-login: Disconnected: Connection closed (no auth attempts in 0 secs): user=<>, rip=127.0.0.1, lip=127.0.0.1, secured, session=<kzz6aQEGKO5/AAAB>
    Sep 23 07:10:03 yakitori dovecot: imap-login: Disconnected: Connection closed (disconnected before auth was ready, waited 0 secs): user=<>, rip=127.0.0.1, lip=127.0.0.1, secured, session=<Hz76aQEGSt1/AAAB>
    Sep 23 07:11:01 yakitori CRON[10001]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:11:01 yakitori CRON[10002]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:11:06 yakitori kernel: [ 5229.803831] [UFW BLOCK] IN=eno1 OUT= MAC=01:00:5e:00:00:01:bc:30:d9:a7:dd:da:08:00 SRC=192.168.0.1 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0x00 TTL=1 ID=42124 DF PROTO=2
    Sep 23 07:12:01 yakitori CRON[10030]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:12:01 yakitori CRON[10031]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:12:21 yakitori named[1060]: connection refused resolving '_.127.bl.blocklist.de/A/IN': 185.21.103.31#53
    Sep 23 07:12:24 yakitori named[1060]: connection refused resolving '_.0.0.127.bl.blocklist.de/A/IN': 198.199.90.210#53
    Sep 23 07:12:25 yakitori named[1060]: connection refused resolving '1.0.0.127.bl.blocklist.de/A/IN': 198.199.90.210#53
    Sep 23 07:12:25 yakitori named[1060]: connection refused resolving '1.0.0.127.bl.blocklist.de/A/IN': 185.21.103.31#53
    Sep 23 07:13:01 yakitori CRON[10078]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:13:01 yakitori CRON[10079]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:13:11 yakitori kernel: [ 5354.805164] [UFW BLOCK] IN=eno1 OUT= MAC=01:00:5e:00:00:01:bc:30:d9:a7:dd:da:08:00 SRC=192.168.0.1 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0x00 TTL=1 ID=55495 DF PROTO=2
    Sep 23 07:13:23 yakitori postfix/anvil[9749]: statistics: max connection rate 1/60s for (smtp:193.42.33.87) at Sep 23 07:09:10
    Sep 23 07:13:23 yakitori postfix/anvil[9749]: statistics: max connection count 1 for (smtp:193.42.33.87) at Sep 23 07:09:10
    Sep 23 07:13:23 yakitori postfix/anvil[9749]: statistics: max cache size 2 at Sep 23 07:09:15
    Sep 23 07:14:01 yakitori CRON[10107]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:14:01 yakitori CRON[10108]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:14:44 yakitori postfix/smtpd[10136]: connect from unknown[147.78.103.52]
    Sep 23 07:14:47 yakitori postfix/smtpd[10136]: warning: unknown[147.78.103.52]: SASL LOGIN authentication failed: UGFzc3dvcmQ6
    Sep 23 07:14:47 yakitori postfix/smtpd[10136]: disconnect from unknown[147.78.103.52] ehlo=1 auth=0/1 quit=1 commands=2/3
    Sep 23 07:15:01 yakitori CRON[10145]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:15:01 yakitori CRON[10144]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 23 07:15:01 yakitori CRON[10146]: (getmail) CMD (/usr/local/bin/run-getmail.sh > /dev/null 2>> /dev/null)
    Sep 23 07:15:02 yakitori pure-ftpd: ([email protected]) [INFO] New connection from 127.0.0.1
    Sep 23 07:15:02 yakitori pure-ftpd: ([email protected]) [INFO] Logout.
    Sep 23 07:15:02 yakitori postfix/smtpd[10136]: connect from localhost.localdomain[127.0.0.1]
    Sep 23 07:15:02 yakitori postfix/smtpd[10136]: lost connection after CONNECT from localhost.localdomain[127.0.0.1]
    Sep 23 07:15:02 yakitori postfix/smtpd[10136]: disconnect from localhost.localdomain[127.0.0.1] commands=0/0
    Sep 23 07:15:02 yakitori dovecot: imap-login: Disconnected: Connection closed (no auth attempts in 0 secs): user=<>, rip=127.0.0.1, lip=127.0.0.1, secured, session=<5l/aewEG3qN/AAAB>
    Sep 23 07:15:02 yakitori dovecot: pop3-login: Disconnected: Connection closed (no auth attempts in 0 secs): user=<>, rip=127.0
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

  3. nzAndy

    nzAndy Member

    I read that, and went part way through the debugging of server actions tutorial, and put a copy of the log in the post after waiting a few minutes ... the rest of the tutorial looks difficult from my inexperienced side, so I thought I might be able to get some direction how to proceed?
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    The read-before-posting guide says this on the topic:

    When you follow the link, it asks you to post the output you get from server.sh after enabling debug mode.
    But what you posted above is not the debug output from server.sh, it looks to me as if you posted /var/log/syslog instead.
     
  5. nzAndy

    nzAndy Member

    I got as far as this, ... what does this mean, I delete the example, and leave everything after this on the same line?
    upload_2023-9-23_15-48-59.png
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    No. Commenting out a config line on Linux means putting a # at the beginning of the line. See screenshot and compare it with what you see in the file, then you can see that the screenshot has a # in front of the line.
     
  7. nzAndy

    nzAndy Member

    thanks, I think i got there in the end?



    Code:
    23.09.2023-08:04 - WARNING - There is already a lockfile set, but no process running with this pid (14122). Continuing.
    PHP Fatal error:  Uncaught Error: Undefined constant "LOGLEVEL_WARNING" in /usr/local/ispconfig/server/plugins-available/mysql_clientdb_plugin.inc.php:283
    Stack trace:
    #0 /usr/local/ispconfig/server/plugins-available/mysql_clientdb_plugin.inc.php(483): mysql_clientdb_plugin->db_insert()
    #1 /usr/local/ispconfig/server/lib/classes/plugins.inc.php(120): mysql_clientdb_plugin->db_update()
    #2 /usr/local/ispconfig/server/mods-available/database_module.inc.php(99): plugins->raiseEvent()
    #3 /usr/local/ispconfig/server/lib/classes/modules.inc.php(302): database_module->process()
    #4 /usr/local/ispconfig/server/lib/classes/modules.inc.php(235): modules->raiseTableHook()
    #5 /usr/local/ispconfig/server/server.php(180): modules->processDatalog()
    #6 {main}
      thrown in /usr/local/ispconfig/server/plugins-available/mysql_clientdb_plugin.inc.php on line 283
     
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    It looks as if you use an old ISPConfig version that is incompatible with the PHP version you have installed now. You should update ISPConfig to 3.2.11.
     
  9. nzAndy

    nzAndy Member

    okay thanks, I'm using Version: 3.2.9 ... I will see how I go with the update, cheers
     
  10. nzAndy

    nzAndy Member

    Brilliant! thanks Till, the latest update cleared the job queue!
     
    Th0m and till like this.

Share This Page