Bug? Jailkit / chroot crashed server

Discussion in 'General' started by grigra, Sep 19, 2009.

  1. grigra

    grigra New Member

    Hi,
    I just want wo point on a serious issue that probably has to do with jailkit aund ISPConfig 3. I found a thread on another forum which describes the problem very clearly (I think this board is the correct one).

    Source: http://ubuntuforums.org/showthread.php?p=7954438

    I have done exactly the same and got the same problem. My server iss running with Debian Lenny and ISPConfig 3.0.1.4...

    Would be glad if anyone can help me on the issue. I still have access on my files through a rescue system.
     
  2. sh1ny

    sh1ny New Member

    Yup i can confirm the issue. To me it happened on Ubuntu 8.04 with the latest jailkit and ISPConfig 3.0.13. The strange thing is it happened when i tried to ( add succeeded ) add a shell user with username. The client name was default, and in the field for the shell username i wrote nothing and it still accepted it, then the problems described above appeared.
     
  3. rozwell

    rozwell New Member

    Solution

    Hi guys, it's my thread =)

    It wasn't the ISPConfig but default Jailkit configuration.
    You can find a simple solution in the same thread:
    http://ubuntuforums.org/showthread.php?p=7954438
    Cheers!
    rozwell

    p.s.
    I'll send a message to Jailkit author to draw attention on this bug(?).
     
    Last edited: Sep 21, 2009
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    Please check the passwd shadow and group file in the chroot env, they should contain the root user and group by default.
     
  5. till

    till Super Moderator Staff Member ISPConfig Developer

    Thats absolutely ok as default is a valid username.
     
  6. xerc

    xerc New Member

    I have the same problem:

    system can't boot: won't find /sbin/init

    can't chroot to system with rescue cd: /bin/bash not found

    Though all files and libs are there. Jailkit/ISPConfig messed up my hole system. Does anybody have a working solution?

    Regards
     
  7. till

    till Super Moderator Staff Member ISPConfig Developer

    Before we can find a solution we will have to find out what happened as neither the ispconfig nor jailkit developers had been able able to reproduce this until now.

    1) What exactly did you do on the server before this happened?

    Please boot the system from a rescue CD, then mount the harddisk and send me a copy of /etc/passwd and /etc/group by email to dev [at] ispconfig [dot] org

    Then chroot to the system and execute the command:

    echo $PATH

    and send me the output.
     
  8. xerc

    xerc New Member

    Hi, I did the following:

    I followed the "lenny ispconfig perfect server" tutorial, did all the steps except installing jailkit. That was 2 weeks ago. Yesterday I installed jailkit 2.8 from source. After that, I created a SSH-User with ISPConfig and selected jailkit.
    I was logged in as root via SSH, but after the ISPConfig Cron ran, I was not able to execute any command, "ANY-COMMAND not found".
    Rebooting doesn't work, /sbin/init not found.

    CHROOT at rescue system doesn't work: /bin/bash not found.
    But /bin/bash and all libs are existant on rescue and crashed system.

    chroot: cannot run command `/bin/bash': No such file or directory

    /etc/passwd

    Code:
    root:x:0:
    daemon:x:1:
    bin:x:2:
    sys:x:3:
    
    SNIP
    
    ssh:x:103:
    ntp:x:104:
    mysql:x:105:
    ssl-cert:x:106:postgres
    postfix:x:107:
    postdrop:x:108:
    clamav:x:109:
    amavis:x:110:clamav
    vmail:x:5000:
    getmail:x:5001:
    sshusers:x:5002:web1,web2,web3,web4,web5,web7,web8,web9,web10,web11,web13,web15,web17,web19,web23,web25
    ispconfig:x:5003:www-data
    client0:x:5004:www-data
    client3:x:5005:www-data
    client1:x:5006:www-data
    postgres:x:111:
    lm_crawler:x:5014:
    client4:x:5015:www-data
    client5:x:5016:www-data
    
    /etc/passwd

    Code:
    root:x:0:0:root:/root:/bin/sh
    daemon:x:1:1:daemon:/usr/sbin:/bin/sh
    bin:x:2:2:bin:/bin:/bin/sh
    sys:x:3:3:sys:/dev:/bin/sh
    sync:x:4:65534:sync:/bin:/bin/sync
    
    SNIP
    
    getmail:x:5001:5001::/etc/getmail:/bin/sh
    ispconfig:x:5002:5003::/usr/local/ispconfig:/bin/sh
    web1:x:5003:5006::/var/www/clients/client1/web1:/bin/false
    web2:x:5004:5006::/var/www/clients/client1/web2:/bin/false
    
     
    Last edited: Oct 13, 2009
  9. till

    till Super Moderator Staff Member ISPConfig Developer

    Ok, the passwd and group files look normal. What is the output of:

    echo $PATH

    after you did a chroot on the rscue system?

    How did you try to reboot? Did you make a hard reset of the server, does it come up again?
     
  10. xerc

    xerc New Member

    Hi till,

    as I already said, chroot to crashed system is not possible.

    root@rescue / # chroot /mnt/
    chroot: cannot run command `/bin/bash': No such file or directory

    I did a strace, I guess this is the problem:

    execve("/bin/bash", ["/bin/bash", "-i"], [/* 18 vars */]) = -1 ENOENT (No such file or directory)

    But /bin/bash and all libs exist on crashed system.

    Anyway, thats the output of export after chroot crashing:

    Code:
    declare -x HOME="/root"
    declare -x LANG="en_US.ISO-8859-15"
    declare -x LOGNAME="root"
    declare -x LS_COLORS="no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.svgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:"
    declare -x LS_OPTIONS="--color=auto -h"
    declare -x MAIL="/var/mail/root"
    declare -x OLDPWD="/mnt/var"
    declare -x PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    declare -x PS1="\\[\\033[01;31m\\]\\u\\[\\033[01;33m\\]@\\[\\033[01;32m\\]\\h \\[\\033[01;33m\\]\\w \\[\\033[01;35m\\]\\\$ \\[\\033[00m\\]"
    declare -x PWD="/mnt"
    declare -x SHELL="/bin/bash"
    declare -x SHLVL="1"
    declare -x SSH_CLIENT="XXX 22"
    declare -x SSH_CONNECTION="XXX 22"
    declare -x SSH_TTY="/dev/pts/0"
    declare -x TERM="xterm"
    declare -x USER="root"
    
    
    
     
    Last edited: Oct 13, 2009
  11. till

    till Super Moderator Staff Member ISPConfig Developer

    May you please check with "ls -la" if /bin/bash and /bin/sh are files or symlinks inside the crashed system. If they are symlinks, please check if the files that the symlinks point to exist.
     
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    To all posters in this thread: May you please confirm to me that you are using jailkit 2.8 on your systems that had these problems.
     
  13. xerc

    xerc New Member

    I used JailKit 2.8

    /bin/bash and /bin/sh were files, no symlinks
     
  14. till

    till Super Moderator Staff Member ISPConfig Developer

    Is this a 32bit or 64bit linux system?

    Below in the quoted area a few questions from Olivier. Please try to run the suggested comands on the binaries from the broken system and report the results.

     
  15. xerc

    xerc New Member

    It is 64 bit.

    All files and all symlinks where correct.

    I'm currently reinstalling the server because we need it so much, could not wait any longer, but I still have the HD Dump.


    file bash
    Code:
    bash: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.8, stripped
    
    static:~/rescue_backup/rescue/bin# ldd bash
    Code:
    static:~/rescue_backup/rescue/bin# ldd bash
            linux-vdso.so.1 =>  (0x00007fffb61fe000)
            libncurses.so.5 => /lib/libncurses.so.5 (0x00007f5aadc8c000)
            libdl.so.2 => /lib/libdl.so.2 (0x00007f5aada88000)
            libc.so.6 => /lib/libc.so.6 (0x00007f5aad735000)
            /lib64/ld-linux-x86-64.so.2 (0x00007f5aadecb000)
    
     
  16. till

    till Super Moderator Staff Member ISPConfig Developer

    I was able to reproduce the problem now. It occurs only on 64Bit systems. I'am just preparing a virtual machine image to send it to Olivier.
     
  17. till

    till Super Moderator Staff Member ISPConfig Developer

    The problem in jailkit has been fixed. The current CVS version of jailkit is working fine for me now and Olivier will release a fixed jailkit version soon. To get the fixed cvs release, use this command:

    Code:
    cvs -z3 -d:pserver:[email protected]:/sources/jailkit co jailkit
    and then run ./configure && make && make install

    inside the jailkit directory.

    If you have a 64Bit test system, it would be great if you might be able to test it. We have also found a typo in the jk_init.ini file which is delivered by ispconfig, this is not related directly to the problem in this thread but as the CVS version of jailkit has a better config file parser, it complains about the syntax error in the old file. This has been fixed in ispconfig SVN too and I attached the fixed file as .zip to this post here too.

    Description of the error and possible solutions to make the system accessible again:

    There was a error in the jailkit scripts in jailkit 2.8 that copy the system librarys which only affects directorys or files that are symlinked. In Debian and Ubuntu 64 bit, the lib64 directory is a symlink to the lib directory which causes the librarys in lib to be oevrwritten or removed. A possible solution to to make a broken system accessible again is to copy the librarys in lib from a working linux system or rescue cd (must be the same version and distribution of course) to your broken system and then reboot.
     

    Attached Files:

  18. sh1ny

    sh1ny New Member

    Thanks, i will test this on a virtual machine soon.
     
  19. till

    till Super Moderator Staff Member ISPConfig Developer

  20. Nikola

    Nikola Member

Share This Page