Jailkit issue with folder removal

Discussion in 'ISPConfig 3 Priority Support' started by variable99, Oct 29, 2025.

  1. variable99

    variable99 Member HowtoForge Supporter

    When jailkit user is removed, the folder structure is left intact. And when jailkit user set up again, it complains like this:
    Log entry for that operation:

    https://pastebin.com/nM5RH9gr

    In short: removing jailkit user leave file / folder "left overs" which can't be removed:

    Folder structure:


    Code:
    drwxr-xr-x 16 root   root      4096 Oct 29 12:43 .
    drwxr-xr-x  3 root   root      4096 Sep  2 14:03 ..
    drwxr-x---  2 web938 client989 4096 Sep  2 14:03 .composer
    drwx------  2 web938 client989 4096 Oct 29 12:48 .ssh
    drwxr-xr-x  3 web938 client989 4096 Oct 10 14:23 backup
    lrwxrwxrwx  1 root   root         7 Oct 29 12:43 bin -> usr/bin
    drwxr-xr-x  2 web938 client989 4096 Sep  2 14:03 cgi-bin
    drwxr-xr-x  2 root   root      4096 Oct 29 12:49 etc
    drwxr-xr-x  5 root   root      4096 Oct 29 12:48 home
    lrwxrwxrwx  1 root   root         7 Oct 29 12:43 lib -> usr/lib
    lrwxrwxrwx  1 root   root         9 Oct 29 12:43 lib64 -> usr/lib64
    drwxr-xr-x  3 root   root      4096 Oct 29 02:03 log
    drwx--x---  2 web938 client989 4096 Sep  2 14:03 private
    drwx------  3 root   root      4096 Oct 29 12:43 root
    drwxr-xr-x  2 root   root      4096 Sep  2 14:03 ssl
    drwxrwx---  2 web938 client989 4096 Oct 10 03:03 tmp
    drwxr-xr-x  2 root   root      4096 Oct 29 12:49 usr
    drwx--x---  5 web938 client989 4096 Oct 13 13:04 web
    drwx--x---  2 web938 client989 4096 Sep  2 14:03 webdav
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    Ok, I'll have to see if I can reproduce this on my systems.

    That's intended; they shall not be removed as the jail belongs to the website and not a specific user.

    You must remove the immutable flag of the root folder of the site before you can delete anything in that website:

    chattr -i /var/www/clients/client989/web938
     
    ahrasis and variable99 like this.
  3. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    I faced this, may be all of us, all the time, when removing shell user, and though a little bit annoying, I am fine with it, but can we do something about it @till? I guess some extra lines in its code would help us clean it up nicely right? At least in one list look at the web site root folder, we simply know shell user is not setup or is already removed.
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    As I mentioned, the jail shall not be removed. This is intentionally and not something that was forgotten.
     
  5. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    I think he and I both mean the same folder "left overs" for jail shell user, not the jail itself, which are still there after the shell user is removed, which you also suggested to use chattr -i before being able to delete them as a solution above mine? My proposal was to automatically remove that same folder "left overs" after the shell user is removed, by adding extra line to ISPConfig current code, that was all. Currently I am fine with it, but think it seems better if they were remove after shell user is removed, a thought that can safely be disregarded.
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    The folders he posted above (like /var/www/clients/client989/web938/bin) are for the jail of the website and not a specific user, and they are also used by other services on the website. As I said, we intentionally are not removing them; you are free to modify the ISPConfig code on your own server and risk breaking your website, that's up to you. We care about ISPConfig users and that their websites do not get messed up, that's why we do not remove them by default. But if you do not have clients and don't care if your website breaks on user removal, feel free to modify the code and remove them. It might work in some cases, in other your site will stop working fully or partially.
     
  7. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    After my posts above, I revisited the code just to make sure and the ISPConfig GUI and make some tests just to get better understanding, to my surprise ISPConfig is already set to cleanup the leftover folders via "Delete unused jailkit chroot" in the option tab if it is ticked or enabled. My previous statements were all wrong as it is already a feature of ISPConfig. Even the bin folder was cleaned up if that option was ticked. :rolleyes:

    Before shell user:
    backup cgi-bin log private ssl tmp web
    Having shell user:
    backup bin cgi-bin dev etc home lib lib64 log private run ssl tmp usr var web
    After user remove (option ticked):
    backup cgi-bin log private ssl tmp web
     
    Last edited: Oct 30, 2025 at 12:49 PM
  8. variable99

    variable99 Member HowtoForge Supporter

    Anyone seeking to solve this problem:
    1. Remove jailkit user (not shell, but jailed SSH user!).
    2. chattr -i /user/folder
    3. unlink lib; unlink bin; unlink bin64 (they are hardlinks)
    4. chattr +i /user/folder
    5. Create new user without problems.

    Hardlinked folder leftovers is the problem as it interferes with user re-creation. If you just create jailed user and never delete it - you never notice this problem. "Delete unused jailkit chroot" does nothing to care about this.

    I will modify ISPC code, but this is not normal.
     
  9. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    I don't think there is a need for any modifications to the code. Enabling "Delete unused jailkit chroot" really clean them up after the shell user is removed whether you opted jailkit or not. If you're still seeing the leftover folders, adjust some of that web site settings so it will resync it individually or use Tool > Resync > Websites which will resync all. Check again after that, they should be cleaned.
     
    Last edited: Nov 1, 2025 at 9:36 AM
    variable99 likes this.
  10. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    don't forget it also creates those jailkit files/folders if you're using a chrooted php-fpm, which i believe is enabled by default, so removing those folders without disabling that will break stuff.
     
  11. variable99

    variable99 Member HowtoForge Supporter

    You were correct. Doing website resync helped to clean folders. But why? Why "Delete unused jailkit" option does not kick-in when I deleting user... Anyway, now when there is viable solution, I will not have to modify system files. Thank you.
     
    ahrasis likes this.
  12. variable99

    variable99 Member HowtoForge Supporter

    Ok, digging a bit deeper into this jailkit phenomenon. Apparently there is cron in /usr/local/ispconfig/server/lib/classes/cron.d/600-jailkit_maintenance.inc.php which has this section:


    PHP:
    elseif ($rec['delete_unused_jailkit'] == 'y') {
                                    
    //$app->log('Removing unused jail: '.$rec['document_root'], LOGLEVEL_DEBUG);
                                    
    print 'Removing unused jail: '.$rec['document_root']."\n";
                                    
    $app->system->web_folder_protection($rec['document_root'], false);
                                    
    $app->system->delete_jailkit_chroot($rec['document_root'], $options);
                                    
    $app->system->web_folder_protection($rec['document_root'], true);

                                    
    $app->db->query("UPDATE `web_domain` SET `last_jailkit_update` = NOW(), `last_jailkit_hash` = NULL WHERE `document_root`>
                            }
    Since all my websites has "Delete unused jailkit" option upon creation - I assume that this portion of code does not fire up. It is only executed once I do website resync via Tools.

    Investigative work led me to something interesting.

    MySQL data on table "sys_cron":
    How's that possible?
     
    ahrasis likes this.
  13. till

    till Super Moderator Staff Member ISPConfig Developer

    You can try using the cron_debug.php script in the server folder to run the plugin manually and see if it throws any errors on your system.
     
    ahrasis likes this.
  14. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    I am interested on debugging results as there may be some jailkit errors instead of cron errors.
     
  15. variable99

    variable99 Member HowtoForge Supporter

    Here is the result:

    Code:
    php cron_debug.php --cronjob=600-jailkit_maintenance.inc.php
    Server time: Mon Nov 3 13:45:14 EET 2025
    PHP time:

    PHP:
    php -'echo trim(shell_exec("date +\"%Y-%m-%d %H:%M:%S\""));'
    2025-11-03 13:46:40

    What else I have observed:

    There is file /usr/local/ispconfig/server/temp/jailkit_force_update.ts CRON touches it to determine last CRON I believe. I have multi-server setup and all related webservers had this file stalled on last year. I have deleted that file and CRON re-created it with date like:

    -rw-r--r-- 1 root root 0 Nov 3 11:50 jailkit_force_update.ts
     
    Last edited: Nov 3, 2025 at 12:52 PM
  16. till

    till Super Moderator Staff Member ISPConfig Developer

    That's ok as ISPConfig uses the time zone as configured in its config file, which is not necessarily your server's time zone, and in your PHP test script, you also did not set a time zone. Besides that, the time does not matter for a manual plugin call anyway, as it always executes the code.
     
    variable99 likes this.
  17. variable99

    variable99 Member HowtoForge Supporter

    Correct, I just posted time output for reference. Where can I find shell user delete action when I do it from control panel? I found form, .tpl etc., but can't find exact function which renders shell account deletion.
     
  18. till

    till Super Moderator Staff Member ISPConfig Developer

    In the shell user and the shell user jailkit plugin of the server part.
     
  19. till

    till Super Moderator Staff Member ISPConfig Developer

    You probably checked the file creation and not the modification time. The Code uses filemtime, which is the time the file was last modified and not the time the file was created.
     
  20. variable99

    variable99 Member HowtoForge Supporter

    Ok, playing with some functions to force cleanup of jailkit user. So far:

    shelluser_jailkit_plugin code:


    PHP:
    if(isset($web['delete_unused_jailkit']) && $web['delete_unused_jailkit'] == 'y') {
                    
    //$this->_delete_jailkit_if_unused($web['domain_id']);
                    
    $app->system->delete_jailkit_chroot($web['document_root'], $options);
                    
    $app->db->query("UPDATE `web_domain` SET `last_jailkit_update` = NOW(), `last_jailkit_hash` = NULL WHERE `document_root` = ?"$web['document_root']);
                }
    I have replaced plugin code with jailkit cleanup cron code portion. But unfortunately it still does not delete left overs.

    Is there any difference from jailkit maintenance cron code:
    PHP:
    elseif ($rec['delete_unused_jailkit'] == 'y') {
                    
    //$app->log('Removing unused jail: '.$rec['document_root'], LOGLEVEL_DEBUG);
                    
    print 'Removing unused jail: '.$rec['document_root']."\n";
                    
    $app->system->web_folder_protection($rec['document_root'], false);
                    
    $app->system->delete_jailkit_chroot($rec['document_root'], $options);
                    
    $app->system->web_folder_protection($rec['document_root'], true);

                    
    $app->db->query("UPDATE `web_domain` SET `last_jailkit_update` = NOW(), `last_jailkit_hash` = NULL WHERE `document_root` = ?"$rec['document_root']);
                }
    And jailkit_plugin code:

    PHP:
    $app->system->web_folder_protection($web['document_root'], false);

                
    $userid intval($app->system->getuid($data['old']['username']));
                
    $command 'killall -u ? ; ';
                
    $command .= 'userdel -f ? &> /dev/null';
                
    $app->system->exec_safe($command$data['old']['username'], $data['old']['username']);

                
    // Remove the jailed user from passwd and shadow file inside the jail
                
    $app->system->removeLine($data['old']['dir'].'/etc/passwd'$data['old']['username'].':');
                
    $app->system->removeLine($data['old']['dir'].'/etc/shadow'$data['old']['username'].':');

                if(@
    is_dir($data['old']['dir'].$jailkit_chroot_userhome)) {
                    
    $this->_delete_homedir($data['old']['dir'].$jailkit_chroot_userhome,$userid,$data['old']['parent_domain_id']);

                    
    $app->log("Jailkit Plugin -> delete chroot home:".$data['old']['dir'].$jailkit_chroot_userhomeLOGLEVEL_DEBUG);
                }

                if(isset(
    $web['delete_unused_jailkit']) && $web['delete_unused_jailkit'] == 'y') {
                    
    $this->_delete_jailkit_if_unused($web['domain_id']);
                }

                
    $app->system->web_folder_protection($web['document_root'], true);
    They are almost the same - and use this system function:

    PHP:
    public function delete_jailkit_chroot($home_dir$options = array()) {
            global 
    $app;

            
    $app->log("delete_jailkit_chroot called for $home_dir with options ".print_r($optionstrue), LOGLEVEL_DEBUG);
            
    $app->uses('ini_parser');

            
    // Disallow operating on root directory
            
    if(realpath($home_dir) == '/') {
                
    $app->log("delete_jailkit_chroot: invalid home_dir: $home_dir"LOGLEVEL_WARN);
                return 
    false;
            }

            if(!
    is_dir($home_dir)) {
                
    $app->log("delete_jailkit_chroot: jail directory does not exist: $home_dir"LOGLEVEL_DEBUG);
                return 
    false;
            }

            
    $jailkit_directories = array(
                
    'bin',
                
    'dev',
                
    'etc',
                
    'lib',
                
    'lib32',
                
    'lib64',
                
    'opt',
                
    'sys',
                
    'usr',
                
    'var',
                
    'run',        # not used by jailkit, but added for cleanup
            
    );

            foreach (
    $options as $opt) {
                switch (
    $opt) {
                default:
                    if (
    preg_match('@^skip[ =]/?(.+)$@'$opt$matches) ) {
                        
    $matches[1] = ltrim($matches[1], '/');
                        if (
    in_array($matches[1], $jailkit_directories)) {
                            
    $app->log("delete_jailkit_chroot: skipping removal of jailkit directory .$home_dir/".$matches[1]
                                . 
    "; if this is in use as a web folder, it is insecure and should be fixed."LOGLEVEL_WARN);
                        }
                        
    $jailkit_directories $app->functions->array_unset_by_value($jailkit_directories$matches[1]);
                    }
                    break;
                }
            }

            
    $removed '';
            foreach (
    $jailkit_directories as $dir) {
                
    $jail_dir rtrim($home_dir'/') . '/'.$dir;

                if (
    is_link($jail_dir)) {
                    
    unlink($jail_dir);
                    
    $removed .= ' /'.$dir;
                } elseif (
    is_dir($jail_dir)) {
                    
    $this->rmdir($jail_dirtrue);
                    
    $removed .= ' /'.$dir;
                }

            }

            
    $app->log("delete_jailkit_chroot: removed from jail $home_dir$removed"LOGLEVEL_DEBUG);

            
    // remove /home if empty
            
    $home rtrim($home_dir'/') . '/home';
            @
    rmdir($home);  # ok to fail if non-empty

            // otherwise archive under /private
            
    $private rtrim($home_dir'/') . '/private';
            if (
    is_dir($home) && is_dir($private)) {
                
    $archive $private.'/home-'.date('c');
                
    rename($home$archive);
            }

            
    // remove $home_dir from /etc/jailkit/jk_socketd.ini
            
    if (is_file('/etc/jailkit/jk_socketd.ini')) {
                
    $jk_socketd_ini $app->ini_parser->parse_ini_file('/etc/jailkit/jk_socketd.ini');
                
    $log $home '/dev/log';
                if (isset(
    $jk_socketd_ini[$log])) {
                    unset(
    $jk_socketd_ini[$log]);
                    
    $app->log('delete_jailkit_chroot: writing /etc/jailkit/jk_socketd.ini'LOGLEVEL_DEBUG);
                    
    $app->ini_parse->write_ini_file($jk_socketd_ini'/etc/jailkit/jk_socketd.ini');
                }
            }

            return 
    true;
        }
     

Share This Page