Jobqueue hang: The following changes are not yet populated to all servers

I tried to edit a client's cron job while logged in as admin but the change did not propagate. I seem to remember reading somewhere that there is a bug that admin cannot create/modify cron jobs for clients, it has to be done within the client account so perhaps this is where I went wrong but whatever it is I cannot get back to a working system now, every job , which is now a major problem, just hangs in the queue whether it is created by admin or client accounts.

I followed the instructions in https://www.howtoforge.com/communit...remove-job-from-job-queue-when-stucked.80951/ to removed all the hung jobs but the problem persists. I cleared it out again and rebooted, but still the same issue.

How do I get the job queue working again ?

PS: I have tried https://www.faqforge.com/linux/debugging-ispconfig-3-server-actions-in-case-of-a-failure/ but changing server loglevel to debug also gets held in the queue.... I really need some help.

pps: Could it be a php issue ?...

Code:

root@VPS1102910604:~# /usr/local/ispconfig/server/server.sh
/usr/local/ispconfig/server/server.sh: line 18: /usr/bin/php: No such file or directory
/usr/local/ispconfig/server/server.sh: line 25: /usr/bin/php: No such file or directory

root@VPS1102910604:~# cat /usr/local/ispconfig/server/server.sh
#!/bin/bash


PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin

. /etc/profile

umask 022

if [ -f /usr/local/ispconfig/server/lib/php.ini ]; then
        PHPINIOWNER=`stat -c %U /usr/local/ispconfig/server/lib/php.ini`
        if [ $PHPINIOWNER == 'root' ] || [ $PHPINIOWNER == 'ispconfig'  ]; then
                export PHPRC=/usr/local/ispconfig/server/lib
        fi
fi

cd /usr/local/ispconfig/server
/usr/bin/php -q \
    -d disable_classes= \
    -d disable_functions= \
    -d open_basedir= \
    /usr/local/ispconfig/server/server.php

cd /usr/local/ispconfig/security
/usr/bin/php -q \
    -d disable_classes= \
    -d disable_functions= \
    -d open_basedir= \
    /usr/local/ispconfig/security/check.php
root@VPS1102910604:~# ls /usr/bin/php -al
ls: cannot access '/usr/bin/php': No such file or directory
root@VPS1102910604:~#
root@VPS1102910604:/bin# php

Command 'php' not found, but can be installed with:

apt install php7.2-cli
apt install hhvm
root@VPS1102910604:/bin# apt list --installed | grep php

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libapache2-mod-php/bionic,bionic,now 1:7.2+60ubuntu1 all [installed]
libapache2-mod-php7.2/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php-apcu/bionic,now 5.1.9+4.0.11-1build1 amd64 [installed]
php-apcu-bc/bionic,now 1.0.3-2ubuntu2 amd64 [installed,automatic]
php-auth-sasl/bionic,bionic,now 1.0.6-3 all [installed,automatic]
php-bz2/bionic,bionic,now 1:7.2+60ubuntu1 all [installed,automatic]
php-common/bionic,bionic,now 1:60ubuntu1 all [installed,automatic]
php-gettext/bionic,bionic,now 1.0.12-0.1 all [installed]
php-imagick/bionic,now 3.4.3~rc2-2ubuntu4 amd64 [installed,upgradable to: 3.4.3~rc2-2ubuntu4.1]
php-mail-mime/bionic,bionic,now 1.10.2-0.1 all [installed,automatic]
php-memcache/bionic,now 3.0.9~20160311.4991c2f-5build2 amd64 [installed]
php-net-sieve/bionic,bionic,now 1.4.1-1 all [installed]
php-net-smtp/bionic,bionic,now 1.8.0-1 all [installed,automatic]
php-net-socket/bionic,bionic,now 1.0.14-2 all [installed,automatic]
php-pear/now 1:1.10.5+submodules+notgz-1ubuntu1.18.04.1 all [installed,upgradable to: 1:1.10.5+submodules+notgz-1ubuntu1.18.04.2]
php-php-gettext/bionic,bionic,now 1.0.12-0.1 all [installed,automatic]
php-phpseclib/bionic,bionic,now 2.0.9-1 all [installed,automatic]
php-soap/bionic,bionic,now 1:7.2+60ubuntu1 all [installed]
php-tcpdf/bionic,bionic,now 6.2.13+dfsg-1ubuntu1 all [installed,automatic]
php-xml/bionic,bionic,now 1:7.2+60ubuntu1 all [installed,automatic]
php7.2/now 7.2.19-0ubuntu0.18.04.2 all [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-bz2/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-cgi/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-cli/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-common/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-curl/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-fpm/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-gd/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-imap/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-intl/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-json/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-mbstring/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-mysql/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-opcache/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-pspell/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-readline/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-recode/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-soap/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-sqlite3/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-tidy/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-xml/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-xmlrpc/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-xsl/now 7.2.19-0ubuntu0.18.04.2 all [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
php7.2-zip/now 7.2.19-0ubuntu0.18.04.2 amd64 [installed,upgradable to: 7.2.24-0ubuntu0.18.04.7]
phpmyadmin/bionic,bionic,now 4:4.6.6-5 all [installed,upgradable to: 4:4.6.6-5ubuntu0.5]

I am going to admit I may have caused this a couple of days ago. After checking history I had mistakenly run a wget via the command line to /usr/bin/php . I realised I had wrongly written the downloaded file to the /usr/bin/php directory and promptly deleted it. But now I'm thinking that may be the root cause of this problem. I still don't know how to fix it though.

 

Hi Thom, yes I did that - the output was:

Code:

root@VPS1102910604:~# /usr/local/ispconfig/server/server.sh
/usr/local/ispconfig/server/server.sh: line 18: /usr/bin/php: No such file or directory
/usr/local/ispconfig/server/server.sh: line 25: /usr/bin/php: No such file or directory

I'm pretty sure the problem is that I accidentally overwrote /usr/bin/php with a bad wget a couple of days ago which is preventing the php commands in server.sh from working. If I'm correct, I just need to restore /usr/bin/php without affecting anything else now.


UPDATE:

Code:

sudo rm -Rf /usr/bin/php
sudo apt-get install --reinstall php7.2-cli

seems to have it working again , phew !