Hi, While creating a cronjob through the ispconfig panel, I noticed that the job was not reflected and remains in the job queue. After creating the cronjob, nothing happened and I tried other changes in the ispconfig panel, but it is not being applied to the server. The contents currently in the job queue are as follows. Delete cron job: 1 Create cron job: 1 Update cron job: 1 Update server settings: 2 Update shell user: 3 Update website settings: 2 My ispconfig version is 3.2.2 (stable), and I reconfigured permissions in master database and reconfigured your services when updating. This is my information. Code: OS: Ubuntu 16.04 PHP CLI version: 7.0 (I tried versions 7.2 and 7.4, but got the same result.) The result of executing /usr/local/ispconfig/server/server.sh to check the debugging log is as follows. Code: 22.01.2021-01:25 - WARNING - There is already a lockfile set, but no process running with this pid (14734). Continuing. 22.01.2021-01:25 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'. 22.01.2021-01:25 - DEBUG - Found 11 changes, starting update process. 22.01.2021-01:25 - DEBUG - Calling function 'insert' from plugin 'cron_jailkit_plugin' raised by event 'cron_insert'. 22.01.2021-01:25 - DEBUG - safe_exec cmd: chattr -i '/var/www/clients/client1/web2' - return code: 0 22.01.2021-01:25 - DEBUG - safe_exec cmd: chattr +i '/var/www/clients/client1/web2' - return code: 0 22.01.2021-01:25 - DEBUG - safe_exec cmd: chattr -i '/var/www/clients/client1/web2' - return code: 0 22.01.2021-01:25 - DEBUG - update_jailkit_chroot called for /var/www/clients/client1/web2 with options Array ( [0] => force ) 22.01.2021-01:25 - DEBUG - update_jailkit_chroot: searching for hardlinks in /var/www/clients/client1/web2/bin 22.01.2021-01:25 - DEBUG - update_jailkit_chroot: searching for hardlinks in /var/www/clients/client1/web2/dev 22.01.2021-01:25 - DEBUG - update_jailkit_chroot: searching for hardlinks in /var/www/clients/client1/web2/etc 22.01.2021-01:25 - DEBUG - removing broken symlink /var/www/clients/client1/web2/lib/x86_64-linux-gnu/libncursesw.so.5 22.01.2021-01:25 - DEBUG - update_jailkit_chroot: searching for hardlinks in /var/www/clients/client1/web2/lib 22.01.2021-01:25 - DEBUG - update_jailkit_chroot: searching for hardlinks in /var/www/clients/client1/web2/lib64 22.01.2021-01:25 - DEBUG - update_jailkit_chroot: searching for hardlinks in /var/www/clients/client1/web2/usr 22.01.2021-01:25 - DEBUG - update_jailkit_chroot: searching for hardlinks in /var/www/clients/client1/web2/var 22.01.2021-01:25 - DEBUG - safe_exec cmd: jk_update --jail='/var/www/clients/client1/web2' --skip=/lib32 --skip=/opt --skip=/sys - return code: 0 22.01.2021-01:25 - DEBUG - jk_update returned: Array ( [0] => skip /var/www/clients/client1/web2/opt/ ) There was no reaction here, so I pressed Ctrl+C to escape. Code: ^CTraceback (most recent call last): File "/usr/sbin/jk_init", line 247, in <module> main() File "/usr/sbin/jk_init", line 244, in main activateConfig(config, jail, args) File "/usr/sbin/jk_init", line 162, in activateConfig ji.handle_cfg_section(config,jail,cfg,section) File "/usr/sbin/jk_init", line 99, in handle_cfg_section self.handle_cfg_section(config,chroot,cfg,tmp) File "/usr/sbin/jk_init", line 108, in handle_cfg_section self.didfiles = jk_lib.copy_binaries_and_libs(chroot, paths2, config['force'], config['verbose'], 1, try_hardlink=config['hardlink'],try_glob_matching=1,handledfiles=self.didfiles) File "/usr/share/jailkit/jk_lib.py", line 633, in copy_binaries_and_libs handledfiles = copy_binaries_and_libs(chroot, ret, force_overwrite, be_verbose, check_libs, try_hardlink=try_hardlink, retain_owner=retain_owner, try_glob_matching=0, handledfiles=handledfiles) File "/usr/share/jailkit/jk_lib.py", line 713, in copy_binaries_and_libs libs = lddlist_libraries(file) File "/usr/share/jailkit/jk_lib.py", line 233, in lddlist_libraries return lddlist_libraries_linux(executable) File "/usr/share/jailkit/jk_lib.py", line 129, in lddlist_libraries_linux p = subprocess.Popen('ldd '+executable, shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True) File "/usr/lib/python2.7/subprocess.py", line 711, in __init__ errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1319, in _execute_child data = _eintr_retry_call(os.read, errpipe_read, 1048576) File "/usr/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call return func(*args) KeyboardInterrupt The changes requested by the panel up to two days ago were well reflected in the server, but it is assumed that the problem occurred after adding the Cron Job in the panel today. The user who created the cron job is an account with a shell user created using jailkit. Thanks.
How long did you wait? The traceback looks like jk_init was running when you hit Ctrl-C. If you run server.sh again I imagine it will attempt to update the same jail (web2) - see what related commands are running when it "hangs" (eg. ps auxww | grep -E 'web2|jk').
Hi, After creating the cron job for the first time, there was no reaction even after waiting about 30 minutes, so I tried additional tasks. The first time I worked on it was 3 hours ago. I ran /usr/local/ispconfig/server/server.sh and waited about 10 minutes, but there was still no response. While there was no response, the currently running process was checked with the command below. Code: ps auxww | grep -E 'web2|jk' The results Code: root 24394 0.0 0.0 26988 6664 pts/2 R+ 02:06 0:00 /usr/bin/python /usr/sbin/jk_init -f -c /etc/jailkit/jk_init.ini -j /var/www/clients/client1/web2 basicshell editors extendedshell netutils ssh sftp scp groups jk_lsh git php composer I didn't feel much when using version 3.1, but it seems that it takes a lot of time to select the Chroot shell as jailkit when creating a shell user after updating to version 3.2. Thanks.
So jk_init hasn't exited after 10 minutes (or hours earlier), it sure shouldn't take that long unless you have extremely slow disk i/o. You could install strace and attach to that process to see if it's doing anything, or what syscall it's waiting on if not (strace -p 24394). Try hitting ctrl-c again, then run the exact same jk_init command from cli as root, see if it prints any errors/clues. What version of jailkit do you have installed? There are bugfixes published for it periodically, if it's pretty old, try installing the latest version and see what happens.
Little progress After posting the first question in this thread, running the server.sh file again, about an hour later, the jailkit files were copied, and then moved on to the next process. However, it is stopped once again in this part. Code: 22.01.2021-02:53 - DEBUG - ssh-rsa setup shelluser_base 22.01.2021-02:53 - DEBUG - ssh-rsa key updated in /var/www/clients/client1/web2/.ssh/authorized_keys 22.01.2021-02:53 - DEBUG - safe_exec cmd: chown -R 'web2':'client1' '/var/www/clients/client1/web2/.ssh' - return code: 0 22.01.2021-02:53 - DEBUG - safe_exec cmd: chmod 600 '/var/www/clients/client1/web2/.ssh/authorized_keys' - return code: 0 22.01.2021-02:53 - DEBUG - safe_exec cmd: chattr +i '/var/www/clients/client1/web2' - return code: 0 22.01.2021-02:53 - DEBUG - Calling function 'update' from plugin 'shelluser_jailkit_plugin' raised by event 'shell_user_update'. 22.01.2021-02:53 - DEBUG - safe_exec cmd: chattr -i '/var/www/clients/client1/web2' - return code: 0 22.01.2021-02:53 - DEBUG - safe_exec cmd: chattr +i '/var/www/clients/client1/web2' - return code: 0 22.01.2021-02:53 - DEBUG - safe_exec cmd: chattr -i '/var/www/clients/client1/web2' - return code: 0 PHP Warning: Invalid argument supplied for foreach() in /usr/local/ispconfig/server/lib/classes/system.inc.php on line 2437 ps auxww | grep -E 'web2|jk' If you check the currently running process using this command, jailkit continues to exist, but the pid is changed every time it is checked, making it difficult to trace with strace. My jailkit version is version 2.19-1 provided by Ubuntu 16.04. It was confirmed by the following method. dpkg -l | grep jailkit Code: ii jailkit 2.19-1 amd64 chroot jail utilities In the middle of writing this answer, all the work in the work queue was done. I'm seeing this as the cause of the slow copying of jailkit. I'm using an SSD, and when I use commands like rsync or zip, it works fast.
Hmm, this should only be called when a jail wasn't already setup (tested by presence of /etc/jailkit directory within the jail). Can you reproduce this? And/or do you still have the debug output that came before that? That's good, and there's no need to run strace if processes are completing/changing (strace was just to troubleshoot why it might be stopped).
For that shell user being created above, what do you have set for 'Base Dir' under the shell user options?
I uncovered one bug though I don't think it's the issue(s) you're having right offhand - when changing the shell user Base dir, in my case from /var/www/clients/client1/web20 to /var/www/clients/client1/web20/blah, I see this error: Code: 21.01.2021-13:08 - DEBUG - Calling function 'update' from plugin 'shelluser_base_plugin' raised by event 'shell_user_update'. 21.01.2021-13:08 - DEBUG - Homedir New: /var/www/clients/client1/web20/blah 21.01.2021-13:08 - DEBUG - Homedir Old: /var/www/clients/client1/web20 21.01.2021-13:08 - DEBUG - safe_exec cmd: chattr -i '/var/www/clients/client1/web20' - return code: 0 PHP Warning: rename(/var/www/clients/client1/web20,/var/www/clients/client1/web20/blah): Invalid argument in /usr/local/ispconfig/server/lib/classes/ system.inc.php on line 894 I'll note when changing the Base Dir I did see a stall in exactly the same place you initially reported having one; I caught the jk_init process still running and was able to tell it was very actively doing it's thing. The timestamps only show minute resolution, and according to that it was less than 1 minute until complete.
Hi, Tested by setting the environment similar to the current server in another server. Same OS, Ubuntu 16.04 Same jailkit settings in ispconfig panel Same jk_init.ini setting It was tested using this configuration. The test server also uses SSD. I used the command below to see which part was the problem. Code: jk_init -c /etc/jailkit/jk_init.ini -f -k -j /var/www/clients/client1/web1565 php Code: Creating symlink /var/www/clients/client1/web1565/usr/share/zoneinfo/posix/Europe/Paris to ../../Europe/Paris Trying to link /usr/share/zoneinfo/Europe/Paris to /var/www/clients/client1/web1565/usr/share/zoneinfo/Europe/Paris The original server takes quite a long time to create each symlink as above. But on the test server, it went very fast. In the beginning, the process of creating symlinks with jailkit was very fast in the original server, but now it is slow. Probably, this is a problem with my server rather than a panel problem. Now, to solve this problem, we need to find out what causes symlinks to slow down. This is my disk usage info. Code: root@new:~# df -h Filesystem Size Used Avail Use% Mounted on udev 32G 0 32G 0% /dev tmpfs 6.3G 59M 6.3G 1% /run /dev/sda1 688G 359G 295G 55% / tmpfs 32G 0 32G 0% /dev/shm tmpfs 5.0M 4.0K 5.0M 1% /run/lock tmpfs 32G 0 32G 0% /sys/fs/cgroup tmpfs 6.3G 0 6.3G 0% /run/user/0 Code: root@new:~# df -i Filesystem Inodes IUsed IFree IUse% Mounted on udev 8227337 398 8226939 1% /dev tmpfs 8232394 1032 8231362 1% /run /dev/sda1 45793280 4366759 41426521 10% / tmpfs 8232394 1 8232393 1% /dev/shm tmpfs 8232394 15 8232379 1% /run/lock tmpfs 8232394 16 8232378 1% /sys/fs/cgroup tmpfs 8232394 4 8232390 1% /run/user/0 Thank you for your help.
Hm. I think I have something similar on my setup (Ubuntu 20.04.1 LTS): Code: Mon 25 Jan 00:16:34 WET 2021 Beginning jailkit maintenance for domain some.domain.name.here at /var/www/clients/client6/web128 Mon 25 Jan 00:16:34 WET 2021 chroot --userspec web128:client6 /var/www/clients/client6/web128 /bin/bash -c true 2>/dev/null Mon 25 Jan 00:16:34 WET 2021 /bin/bash test failed, forcing update Mon 25 Jan 00:16:47 WET 2021 Traceback (most recent call last): Mon 25 Jan 00:16:47 WET 2021 File "/usr/sbin/jk_update", line 290, in <module> Mon 25 Jan 00:16:47 WET 2021 main() Mon 25 Jan 00:16:47 WET 2021 File "/usr/sbin/jk_update", line 272, in main Mon 25 Jan 00:16:47 WET 2021 if (not config.has_key('hardlink')): Mon 25 Jan 00:16:47 WET 2021 AttributeError: 'dict' object has no attribute 'has_key' Mon 25 Jan 00:17:01 WET 2021 finished server.php. This is something that popped up just after midnight, after I've noticed a surge in CPU load. And, indeed, all jk_* functions seem to be teeming with python errors, e.g: Code: # jk_list Traceback (most recent call last): File "/usr/sbin/jk_list", line 159, in <module> main() File "/usr/sbin/jk_list", line 156, in main printResults(results,wide) File "/usr/sbin/jk_list", line 92, in printResults results.sort() TypeError: '<' not supported between instances of 'ListResult' and 'ListResult' This one is even more cryptic: Code: # jk_check ERROR: user web1 does not exist in /var/www/clients/client1/web1//etc/passwd ERROR: user web2 does not exist in /var/www/clients/client2/web2//etc/passwd ERROR: user web6 does not exist in /var/www/clients/client4/web6//etc/passwd ERROR: user web9 does not exist in /var/www/clients/client5/web9//etc/passwd ERROR: user web11 does not exist in /var/www/clients/client6/web11//etc/passwd ERROR: user web12 does not exist in /var/www/clients/client6/web12//etc/passwd ERROR: user web39 does not exist in /var/www/clients/client13/web39//etc/passwd ERROR: user web61 does not exist in /var/www/clients/client6/web61//etc/passwd ERROR: user web69 does not exist in /var/www/clients/client6/web69//etc/passwd ERROR: user web105 does not exist in /var/www/clients/client3/web105//etc/passwd ERROR: user web122 does not exist in /var/www/clients/client7/web122//etc/passwd ERROR: user web123 does not exist in /var/www/clients/client22/web123//etc/passwd ERROR: user web124 does not exist in /var/www/clients/client23/web124//etc/passwd ERROR: user web126 does not exist in /var/www/clients/client3/web126//etc/passwd ERROR: user web127 does not exist in /var/www/clients/client3/web127//etc/passwd ERROR: user web128 does not exist in /var/www/clients/client6/web128//etc/passwd ERROR: user web129 does not exist in /var/www/clients/client6/web129//etc/passwd ERROR: user web130 does not exist in /var/www/clients/client6/web130//etc/passwd ERROR: user web131 does not exist in /var/www/clients/client24/web131//etc/passwd ERROR: user web137 does not exist in /var/www/clients/client25/web137//etc/passwd ERROR: cannot lstat() /home/testchroot cannot check files in /home/testchroot/, it does not exist ERROR: cannot read /var/www/clients/client1/web1/usr/share/terminfo/r/rxvt-unicode-256color or /usr/share/terminfo/r/rxvt-unicode-256color ! ERROR: cannot read /var/www/clients/client1/web1/usr/share/terminfo/s/screen.xterm-256color or /usr/share/terminfo/s/screen.xterm-256color ! ERROR: cannot read /var/www/clients/client1/web1/usr/share/terminfo/s/screen-bce.mlterm or /usr/share/terminfo/s/screen-bce.mlterm ! ERROR: cannot read /var/www/clients/client1/web1/usr/share/terminfo/t/tmux or /usr/share/terminfo/t/tmux ! ERROR: cannot read /var/www/clients/client1/web1/usr/share/terminfo/t/tmux-256color or /usr/share/terminfo/t/tmux-256color ! ERROR: cannot read /var/www/clients/client1/web1/usr/share/vim/vim74/debian.vim or /usr/share/vim/vim74/debian.vim ! ERROR: cannot read /var/www/clients/client1/web1/usr/share/vim/vim74/doc/README.Debian or /usr/share/vim/vim74/doc/README.Debian ! ERROR: cannot read /var/www/clients/client1/web1/usr/share/vim/vim74/doc/help.txt or /usr/share/vim/vim74/doc/help.txt ! ERROR: cannot read /var/www/clients/client1/web1/usr/share/vim/vim74/doc/tags or /usr/share/vim/vim74/doc/tags ! ERROR: cannot read /var/www/clients/client1/web1/usr/lib/libdns.so.95.1.1 or /usr/lib/libdns.so.95.1.1 ! ERROR: symlinks /var/www/clients/client1/web1/usr/lib/libbind9.so.90 and /usr/lib/libbind9.so.90 point to different files! ERROR: cannot read /var/www/clients/client1/web1/usr/lib/libisc.so.92 or /usr/lib/libisc.so.92 ! ERROR: symlinks /var/www/clients/client1/web1/usr/lib/libisccfg.so.90 and /usr/lib/libisccfg.so.90 point to different files! ERROR: cannot read /var/www/clients/client1/web1/usr/lib/liblwres.so.90.0.1 or /usr/lib/liblwres.so.90.0.1 ! ERROR: cannot read /var/www/clients/client1/web1/usr/lib/libisccfg.so.90.0.2 or /usr/lib/libisccfg.so.90.0.2 ! ERROR: cannot read /var/www/clients/client1/web1/usr/lib/libdns.so.95 or /usr/lib/libdns.so.95 ! ERROR: cannot read /var/www/clients/client1/web1/usr/lib/libisc.so.92.1.0 or /usr/lib/libisc.so.92.1.0 ! ERROR: cannot read /var/www/clients/client1/web1/usr/lib/x86_64-linux-gnu/libisc.so.160.0.0 or /usr/lib/x86_64-linux-gnu/libisc.so.160.0.0 ! [...] All this is a bit weird. I'm not really proficient in python development, but it looks like a wrong/missing library is being called at some point...?
I think (not positive) that is an error with a python 2 program running under python 3. Are you using Jailkit 2.21?
Curious; I don't have many ideas right off, short of digging through the code to see exactly what it's doing there. You could attach (to a running process) with or run under strace real quick and see if it's stuck on some system call. Or do the same with ltrace and it will give even more info from libraries. You could check logfiles for errors, too, who knows what might turn up. Filesystem corruption? Apparmor interfering?
Hi, https://www.howtoforge.com/community/threads/jailkit-problems.80852/ Referencing this thread, I thought that Linux's kernel could be causing this problem. Ubuntu 16.04's linux-image-generic version was 4.4.0. I proceeded with the planned Ubuntu release update. I wanted jailkit's problem to be solved. As the Ubuntu version was updated from 16.04 to 20.04, the version of linux-image-generic was changed to 5.4.0, and when I tried jailkit, I noticed that a symbolic link was created very quickly. Thanks for your help!
Hello, I have a same problem. Queue is high Code: /usr/local/ispconfig/server/server.sh 29.03.2024-09:38 - WARNING - There is already a lockfile set, but no process running with this pid (1982). Continuing. PHP Warning: Trying to access array offset on value of type null in /usr/local/ispconfig/server/plugins-available/bind_dlz_plugin.inc.php on line 158 PHP Warning: Trying to access array offset on value of type null in /usr/local/ispconfig/server/plugins-available/bind_dlz_plugin.inc.php on line 158 PHP Warning: Trying to access array offset on value of type null in /usr/local/ispconfig/server/plugins-available/bind_dlz_plugin.inc.php on line 158 PHP Warning: Trying to access array offset on value of type null in /usr/local/ispconfig/server/plugins-available/bind_dlz_plugin.inc.php on line 158 PHP Warning: Trying to access array offset on value of type null in /usr/local/ispconfig/server/plugins-available/bind_dlz_plugin.inc.php on line 158 PHP Fatal error: Uncaught mysqli_sql_exception: UPDATE command denied to user 'ispconfig'@'localhost' for table `named`.`records` in /usr/local/ispconfig/server/lib/classes/db_mysql.inc.php:302 Stack trace: #0 /usr/local/ispconfig/server/lib/classes/db_mysql.inc.php(302): mysqli_query() #1 /usr/local/ispconfig/server/lib/classes/db_mysql.inc.php(339): db->_query() #2 /usr/local/ispconfig/server/plugins-available/bind_dlz_plugin.inc.php(158): db->query() #3 /usr/local/ispconfig/server/lib/classes/plugins.inc.php(120): bind_dlz_plugin->soa_update() #4 /usr/local/ispconfig/server/mods-available/dns_module.inc.php(104): plugins->raiseEvent() #5 /usr/local/ispconfig/server/lib/classes/modules.inc.php(302): dns_module->process() #6 /usr/local/ispconfig/server/lib/classes/modules.inc.php(235): modules->raiseTableHook() #7 /usr/local/ispconfig/server/server.php(180): modules->processDatalog() #8 {main} thrown in /usr/local/ispconfig/server/lib/classes/db_mysql.inc.php on line 302 Where is a problem?
https://forum.howtoforge.com/threads/ispconfig-3-2-9p1-pending-updates-stuck.90479/ Please search first for the error message before posting
