System do not boot up after update

Discussion in 'Installation/Configuration' started by Captain, Jun 12, 2014.

  1. Captain

    Captain Member

    Hello!

    After ISPConfig update to 3.0.5.4p1 and kernel update for 12.04.04LTS to 3.2.0-63-generic
    system do to boot up.
    Problem was in fstab. ISPconfig add lines like:
    Code:
    /var/log/ispconfig/httpd/domain.com /var/www/clients/client23/web11/log    none    bind,nobootwait    0 0
    
    I can boot system only in recovery mode, after fsck.

    I comment all this lines and now system boot up without problem.

    I try to boot from old kernels but still has problem.

    Some one have same problem?

    Falko and Till can you explain this situation?

    Thank you.
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    Check your syslog for errors to see why the booting halts. I never had this problem on a server yet and the lines contain the "nobootwait" option, so they cant block booting even if a directory is missing on your server or not mounted.

    Do you have any directores mounted with nfs or another betwork filesystem?
     
  3. Captain

    Captain Member

    I do not see anything in syslog, system do not write log to syslog only when it will be mounted and system boot up.

    When system do not boot up on screen I see:

    Code:
    stopping read required files in advance
    stopping read required files in advance
    stopping read required files in advance
    stopping read required files in advance
    
    Here is fstab:
    Code:
    # /etc/fstab: static file system information.
    #
    # Use 'blkid' to print the universally unique identifier for a
    # device; this may be used with UUID= as a more robust way to name devices
    # that works even if disks are added and removed. See fstab(5).
    #
    # <file system> <mount point>   <type>  <options>       <dump>  <pass>
    proc            /proc           proc    nodev,noexec,nosuid 0       0
    # / was on /dev/sda5 during installation
    UUID=cd08a4ef-f8d6-4f02-b534-fe31ccc073e7 /               ext4    errors=remount-ro,usrjquota=quota.user,grpjquota=quota.group,jqfmt=vfsv0 0       1
    # /boot was on /dev/sda1 during installation
    UUID=53e0307e-1fbf-43e3-8fb2-7949af72c5cd /boot           ext4    defaults        0       2
    # /tmp was on /dev/sda7 during installation
    UUID=04ea320e-34d3-4ff7-8641-9a73cc7996b0 /tmp            ext2    nodev,nosuid    0       2
    # /var was on /dev/sda6 during installation
    UUID=93ad8100-c99a-4bea-9a1b-c6cebacc45f1 /var            ext4    nodev,nosuid,usrjquota=quota.user,grpjquota=quota.group,jqfmt=vfsv0    0       2
    # swap was on /dev/sda8 during installation
    UUID=0dbc594b-153a-47ce-a268-6e308d3af64d none            swap    sw              0       0
    #/var/log/ispconfig/httpd/com1.com /var/www/clients/client23/web11/log    none    bind,nobootwait    0 0
    #/var/log/ispconfig/httpd/com2.com /var/www/clients/client2/web94/log    none    bind,nobootwait    0 0
    #/var/log/ispconfig/httpd/com3.com /var/www/clients/client24/web89/log    none    bind,nobootwait    0 0
    #/var/log/ispconfig/httpd/com4.com /var/www/clients/client17/web62/log    none    bind,nobootwait    0 0
    #/var/log/ispcon
    
    I have mounted usb ntfs drive for backups. But I unplug it and still has problem.
     
  4. Captain

    Captain Member

    Any ideas Till?
     
  5. till

    till Super Moderator Staff Member ISPConfig Developer

    do both directories of the mountpoints exist. e.g.:

    /var/log/ispconfig/httpd/com1.com

    and

    /var/www/clients/client23/web11/log
     
  6. Captain

    Captain Member

    I checked. All clients have log directory and in /var/log/ispconfig/httpd/ exist all webs.
    But some of them is disabled in ISPConfig (Locked and Canceled).
    It is production server and I can not restart to check may be problem in Locked users in fstab.
     
  7. till

    till Super Moderator Staff Member ISPConfig Developer

    This does not matter as it has no influence on the directory structure or mountpoints.
     
  8. Captain

    Captain Member

    I don't know what to do now. All clients do not have log files now. It is not mounted to his log directories.
    If I enable it in fstab, I dont know what will be after restart.
     
  9. Captain

    Captain Member

    Till,
    It is possible to remount it manually? When system is on. Because all clients do not see their log files.

    Thank you.
     
  10. till

    till Super Moderator Staff Member ISPConfig Developer

    Yes, you can remount it by running:

    mount -a

    command.
     
  11. PaulD

    PaulD New Member

    I have the same problem as this, on 2 separate Ubuntu 12.04 servers, anyone worked out what exactly causes this, at the moment I have all the lines for mounting log directories in fstab commented out.

    I can survive without the log files mounted, is there any way I can stop ispconfig from changing the fstab file apart from the obvious removing write access?
     
  12. bunderthebridge

    bunderthebridge New Member

    I'm running into the same issue with 3 servers running 12.04. The strange thing is that it seems to only be a problem on some sites.
    Short term fix on my production servers was to boot into recovery mode, start networking, then resume boot. All folders from the fstab file were mounted correctly.
    For the one that wasn't in production, I did some experimentation and found that 2 of 6 fstab entries were causing the system to hang on boot. With either of the two enabled, the system would hang on boot some of the time, but sometimes would boot without an issue. Typically, it booted once with the fstab entries enabled but would fail the second time around. When these two were both commented out, I could reboot 5-6 times in a row...
     
  13. till

    till Super Moderator Staff Member ISPConfig Developer

    Some older Ubuntu systems have a bug in the implementation of the _netboot option for filesystems which is used by ISPConfig to prevent boot failures, but on some older Ubuntu versions this may cause the opposite. Remove the _netboot option from all lines in /etc/fstab and disable it in ispconfig under System > server config > web.

    Or alternatively, update to a recent ubuntu version.
     
  14. bunderthebridge

    bunderthebridge New Member

    Thanks till. Apologies if I'm being dense here, but I don't see "_netboot" in fstab. I do, however, see "_netdev." Is that what you meant? I tried removing these and it seems to have addressed the problem, but wanted to double check.

    Similarly, I don't see anything that fits the description under System > server config > web besides Network Filesystem, which was already disabled in my case. Is there a setting that will keep "_netdev" from being included in the fstab entry when new sites are created?

    Thanks much!
     
    Last edited: Apr 22, 2015
  15. till

    till Super Moderator Staff Member ISPConfig Developer

    Yes, thats what I meant.
     
  16. jeppunen

    jeppunen New Member

    I have had this problem with Ubuntu 12.04 couple of years, even currently I'm running Ubuntu 12.04.5 LTS, updated to kernel 3.13.0-65 and with all available patches apt can find. Of course this is a big problem with production server with couple of hundreds of sites inside, but I have learned to live with it and do all reboot requiring improvements at nighttime (which, of course is not optimal for me).

    Previously I also have been using "recovery mode with network" trick (boot to recovery mode, choose network which mounts some parts of the filesystem and continue normal booting). But this does not work always, sometimes I have to go to recovery mode console, edit /etc/fstab and comment out swap row and do all sorts of magic tricks before finally I get the server to boot.

    The problem is definately with _netdev options inside fstab. Nowadays I usually remove _netdev options with sed command before booting and the server boots flawlesly. This is not optimal solution because if the server reboots because of some sort of technical problem, it won't go online until someone goes to recovery console. One solutions might be to add the sed command to crontab and get the cron to remove netdev's like every hour or so.

    The sed command I use is:
    sed -i 's|,_netdev||g' /etc/fstab

    I have restored a full backup of the server to insulated laboratory and upgraded lab-server to Ubuntu 14.04. The server boots just fine with 14.04 even there are _netdev options in fstab. So this problem seems to be with Ubuntu's earlier versions.

    I hope this post helps some other ISPConfig users with this same problem. As an sysadmin, it's not fun to boot production server (with paying customers and SLA's) and realize that the server is not going to go back online and there seem to be no any reason for the problem. What a nightmare for some junior level sysadmin.
     
  17. till

    till Super Moderator Staff Member ISPConfig Developer

    That the _netdev option does not work is an issue in Ubuntu 12.04 only, Ubuntu has fixed this flaw in newer releases. Beside that, _netdev is configurable for years now in ISPConfig System settings, so if you use Ubuntu 12.04 then you can disable it under system > server config > web, no need to use sed for that.
     
  18. jeppunen

    jeppunen New Member

    The only option that I have noticed under web tab is "Network Filesystem", but that is already untagged. Is it this the option you are referring?
     
  19. till

    till Super Moderator Staff Member ISPConfig Developer

    Yes, thats the option. If it is uncheckked then the _netdev option is not set.
     
  20. jeppunen

    jeppunen New Member

    Ok, that option is and has been always unchecked but I still get the _netdev option to new sites.. I also tried to check, save, uncheck and save and create a new site. But still I can see _netdev at the end of the new row on fstab?!? A bug? I'm running the latest p8.
     

Share This Page