Errors after upgrading from Wheezy to Jessie

Discussion in 'ISPConfig 3 Priority Support' started by MaxT, May 23, 2017.

  1. MaxT

    MaxT Active Member HowtoForge Supporter

    I have upgraded from Wheezy to Jessie. (also from Apache 2.2 to Apache 2.4)

    All is working but I have found these persistent Apache errors.
    Please, Can somebody confirm if the cause can be the old Apache syntax?:
    Code:
    # service apache2 restart
    Job for apache2.service failed. See 'systemctl status apache2.service' and 'journalctl -xn' for details.
    
    # systemctl status apache2.service
    ? apache2.service - LSB: Apache2 web server
      Loaded: loaded (/etc/init.d/apache2)
      Drop-In: /lib/systemd/system/apache2.service.d
      +-forking.conf
      Active: failed (Result: exit-code) since mar 2017-05-23 12:29:10 CEST; 18s ago
      Process: 3668 ExecStart=/etc/init.d/apache2 start (code=exited, status=1/FAILURE)
    
    may 23 12:28:50 host.myhost.com apache2[3668]: Starting web server: apache2httpd (pid 1577) already running
    may 23 12:29:10 host.myhost.com apache2[3668]: failed!
    may 23 12:29:10 host.myhost.com apache2[3668]: The apache2 instance did not start within 20 seconds. Please read the log files to discover problems ... (warning).
    may 23 12:29:10 host.myhost.com systemd[1]: apache2.service: control process exited, code=exited status=1
    may 23 12:29:10 host.myhost.com systemd[1]: Failed to start LSB: Apache2 web server.
    may 23 12:29:10 host.myhost.com systemd[1]: Unit apache2.service entered failed state.
    
    # /var/log/apache2/error.log
    [Tue May 23 12:27:02.003960 2017] [access_compat:error] [pid 6334] [client 127.0.0.1:39234] AH01797: client denied by server configuration: /var/www/html/
    
    I have the compatibility module enabled but I'm not sure if still I should replace all the "Order Deny,Allow" syntax in the .conf apache files to end errors. There are a lot!
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    Have you tested the syntax of the apache config with:

    apache2ctl -t

    To see what's wrong?

    Mixed apache 2.2 and 2.4 syntax is a problem as when you e.g. deny access in old syntax, then you can not allow it later in new syntax. So I would replace all old syntax with the new one by:

    1) Updating ISPCOnfig with 'reconfigure" services after the Debian dist upgrade.
    2) In case that apache does not start, remove all symlinks to websites in /etc/apache2/sites-enabled/ except of the symlinks for the ISPConfig vhost and conf files. Then login to ISPConfig, go to Tools > resync, select websites and resync them. This forces ispconfig to write new apache vhost config files for all sites in new apache 2.4 syntax.
     
  3. MaxT

    MaxT Active Member HowtoForge Supporter

    Hi!

    the # apache2ctl -t doesn't show errors.
    After the Jessie update I had fixed some errors with php_gettext, and now apache restarts and the websites are working. However, these errors are still present. And I don't know exactly how to fix them.

    I will do the ISPC update as you says.

    From what you says, I understand that I should do the ISCP update by rewriting the services questions in the process (passwords, etc).
    And after that, then I should do the 2nd step (Tools > resync, select websites and resync them.)
    That's right?.

    thanks!
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    Yes. An ISPConfig update with "reconfgure services = yes" is always required after a dist upfgrade as ISPConfig writes the config files for the installed software versions, so a wheezy system gets different config files then a jessie system. And then use tools > resync to update all website files. Removing the symlinks is not required as apache start, so you can login to ispconfig for the resync.
     
  5. MaxT

    MaxT Active Member HowtoForge Supporter

    process is finished but no success at all...
    At first there were complaints over NameVirtualHost:

    Code:
    # service apache2 restart
    Job for apache2.service failed. See 'systemctl status apache2.service' and 'journalctl -xn' for details.
    
    # systemctl status apache2.service
    ? apache2.service - LSB: Apache2 web server
      Loaded: loaded (/etc/init.d/apache2)
      Drop-In: /lib/systemd/system/apache2.service.d
      +-forking.conf
      Active: failed (Result: exit-code) since mar 2017-05-23 16:04:03 CEST; 8s ago
      Process: 29860 ExecStart=/etc/init.d/apache2 start (code=exited, status=1/FAILURE)
    
    may 23 16:03:43 host.myhost.com apache2[29860]: Starting web server: apache2AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/apache2/sites-enabled/000-ispconfig.conf:72
    may 23 16:03:43 host.myhost.com apache2[29860]: httpd (pid 1577) already running
    may 23 16:04:03 host.myhost.com apache2[29860]: failed!
    may 23 16:04:03 host.myhost.com apache2[29860]: The apache2 instance did not start within 20 seconds. Please read the log files to discover problems ... (warning).
    may 23 16:04:03 host.myhost.com systemd[1]: apache2.service: control process exited, code=exited status=1
    may 23 16:04:03 host.myhost.com systemd[1]: Failed to start LSB: Apache2 web server.
    may 23 16:04:03 host.myhost.com systemd[1]: Unit apache2.service entered failed state.
    
    I have commented all them.
    Also, all the sites inside /etc/apache2/sites-available/* were with error files (*.vhost.err). I have renamed all them and delete the old ones.

    Then restarting apache. Those previous errors were solved but still there is the error about the long time to restart:

    Code:
    # systemctl status apache2.service
    ? apache2.service - LSB: Apache2 web server
      Loaded: loaded (/etc/init.d/apache2)
      Drop-In: /lib/systemd/system/apache2.service.d
      +-forking.conf
      Active: failed (Result: exit-code) since mar 2017-05-23 16:31:42 CEST; 2min 2s ago
      Process: 989 ExecStart=/etc/init.d/apache2 start (code=exited, status=1/FAILURE)
    
    may 23 16:31:22 host.myhost.com apache2[989]: Starting web server: apache2httpd (pid 1577) already running
    may 23 16:31:42 host.myhost.com apache2[989]: failed!
    may 23 16:31:42 host.myhost.com apache2[989]: The apache2 instance did not start within 20 seconds. Please read the log files to discover problems ... (warning).
    may 23 16:31:42 host.myhost.com systemd[1]: apache2.service: control process exited, code=exited status=1
    may 23 16:31:42 host.myhost.com systemd[1]: Failed to start LSB: Apache2 web server.
    may 23 16:31:42 host.myhost.com systemd[1]: Unit apache2.service entered failed state.
    
    the log:

    Code:
    # /var/log/apache2/error.log
    
    [Tue May 23 16:33:01.428407 2017] [access_compat:error] [pid 29852] [client 127.0.0.1:41463] AH01797: client denied by server configuration: /var/www/html/
    [Tue May 23 16:34:02.138632 2017] [access_compat:error] [pid 17680] [client 127.0.0.1:41470] AH01797: client denied by server configuration: /var/www/html/
    [Tue May 23 16:35:01.936689 2017] [access_compat:error] [pid 1672] [client 127.0.0.1:41477] AH01797: client denied by server configuration: /var/www/html/
    [Tue May 23 16:35:02.683285 2017] [access_compat:error] [pid 29852] [client 127.0.0.1:41483] AH01797: client denied by server configuration: /var/www/html/
    
    not sure about the cause of the delay. Maybe a permissions problem... I have this:
    Code:
    # lsattr /var/www
    -------------e-- /var/www/conf
    -------------e-- /var/www/html
    -------------e-- /var/www/clients
    -------------e-- /var/www/php-fcgi-scripts
    -------------e-- /var/www/webalizer
    -------------e-- /var/www/apps
    -------------e-- /var/www/index.html
    
    I'm lost with this. The apache information on errors is really poor :(
    apachectl configtest doesn't show errors and all the websites are working.

    Can you give me a little more help, please?

    thank so much,
     
  6. MaxT

    MaxT Active Member HowtoForge Supporter

    it goes worse.. After rebooting the server the Apache is down. And it cannot go up doing #service apache2 restart:

    Code:
     # service apache2 status
    ? apache2.service - LSB: Apache2 web server
      Loaded: loaded (/etc/init.d/apache2)
      Drop-In: /lib/systemd/system/apache2.service.d
      +-forking.conf
      Active: inactive (dead) since mar 2017-05-23 17:12:08 CEST; 18s ago
      Process: 3477 ExecStop=/etc/init.d/apache2 stop (code=exited, status=0/SUCCESS)
      Process: 3461 ExecStart=/etc/init.d/apache2 start (code=exited, status=0/SUCCESS)
    
    may 23 17:12:08 host.myhost.com apache2[3461]: Starting web server: apache2(98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:443
    may 23 17:12:08 host.myhost.com apache2[3461]: no listening sockets available, shutting down
    may 23 17:12:08 host.myhost.com apache2[3461]: AH00015: Unable to open logs
    may 23 17:12:08 host.myhost.com apache2[3461]: Action 'start' failed.
    may 23 17:12:08 host.myhost.com apache2[3461]: The Apache error log may have more information.
    may 23 17:12:08 host.myhost.com apache2[3461]: .
    may 23 17:12:08 host.myhost.com apache2[3477]: Stopping web server: apache2.
    may 23 17:12:08 host.myhost.com systemd[1]: Started LSB: Apache2 web server.
     
  7. till

    till Super Moderator Staff Member ISPConfig Developer

    This is probably the same reason why ISPConfig saved all vhosts with .err ending as it was not able to restart apache as well. The error 'Address already in use' means that there is already a software running on one of the ports that apache uses. Check that with:

    netstat -ntap

    Or you have multiple 'Listen' statements for the same port somewhere in your apache config. You can check that e.g. with:

    grep -R 'Listen' /etc/apache2/

    there should be no duplicate Listen lines for the same port number.

    And Regarding the namevirtualhost lines that you removed earlier, these were just notices and not harmful, so you could have left them there.
     
  8. MaxT

    MaxT Active Member HowtoForge Supporter

    with netstat -ntap there is no any apache port taken. Surely the problem is what you say about "Listen" duplicates because this is an heritage of a previous mess in my Wheezy.
    The problem is that I don't have a clean model to correct all this.

    Please, Can you point me what is wrong?. These are all the occurrences of "Listen" inside /etc/apache2:

    I believe with this help it can be solved.
    Code:
    # /etc/apache2/sites-enabled/000-ispconfig.conf  (at the end:)
    Listen *:80
    Listen *:443
    Listen *:8080
    
    
    # /etc/apache2/sites-enabled/000-apps.vhost
    Listen 8081
    
    
    # /etc/apache2/ports.conf
    Listen *:80
    <IfModule mod_ssl.c>
      # If you add NameVirtualHost *:443 here, you will also have to change
      # the VirtualHost statement in /etc/apache2/sites-available/default-ssl
      # to <VirtualHost *:443>
      # Server Name Indication for SSL named virtual hosts is currently not
      # supported by MSIE on Windows XP.
    
    Listen *:8081
    Listen *:8080
    Listen *:443
    </IfModule>
    
    <IfModule mod_gnutls.c>
    Listen 443
    </IfModule>
    
    Listen 443
    
    
    # /etc/apache2/sites-available/apps.vhost
    Listen 8081
    
    
    # /etc/apache2/ports.conf.dpkg-dist
    Listen 80
    
    <IfModule ssl_module>
      Listen 443
    </IfModule>
    
    <IfModule mod_gnutls.c>
      Listen 443
    </IfModule>
    
    
    I see the duplicates but the problem is that I don't know the exact place for the "Listen" sentences
     
  9. till

    till Super Moderator Staff Member ISPConfig Developer

    I wonder where you got the Listen lines in the ispconfig.conf file from as the ones that ship with ISPConfig do not add Listen lines.

    Comment out all listen lines in /etc/apache2/sites-enabled/000-ispconfig.conf by adding a # in front.
    Comment out the Listen line for port 8081 in /etc/apache2/ports.conf and then try to start apache again.
     
  10. MaxT

    MaxT Active Member HowtoForge Supporter

    it works!. Thank you :)
    So the ports.conf is the only place for the Listen directive, I suppose.

    Still there is the delay error in restarting Apache, although maybe the cause are the .htaccess with old syntax.

    well at least it works again. Thanks!
     
  11. MaxT

    MaxT Active Member HowtoForge Supporter

    I have discovered the delay error was caused by Modsec Rules. I have commented all provisionally.

    Now there are only 2 errors which are repeated many times and I cannot find the solution (Apache information on errors it's terrible):
    Code:
    [Tue May 23 23:07:02.191105 2017] [authz_core:error] [pid 9330] [client 127.0.0.1:46857] AH01630: client denied by server configuration: /var/www/html/
    
    PHP Fatal error:  xc_fcntl_create: open(/var/www/clients/client2/web8/tmp/.xcache.5011.11748.1.mutex, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
    
    - the first one it's an authz_core:error. I have replaced all the "Deny, allow..." with new 2.4 in all the .htaccess and still appears.
    I wonder if maybe are the .htaccess inside /stats directories, although these seem to be 2.4 compatible. That's right?:
    Code:
    # cat /var/www/clients/client2/web4/web/stats/.htaccess
    AuthType Basic
    AuthName "Members Only"
    AuthUserFile /var/www/clients/client2/web4/web/stats/.htpasswd_stats
    require valid-user
    

    - the second one PHP error seems to be a problem of XCache to write in the /tmp of every website. I have the /tmp's permissions with 770, and it doesn't allow a change to 777 because according ISPC tutorials, it is fixed with chattr to avoid security problems.
    What's the way to fix this error without changing the ISPC security ? (the PHP is Fast-CGI)

    thanks for any help, this upgrade is more painful of what I thought
     
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    Check the directory /var/www/html/, if it does not exist, create it. If it is empty, add a index.html file.

    Which user and group owns the tmp directory and do you have the suexec checkbox enabled in the website settings of the sites?
     
  13. MaxT

    MaxT Active Member HowtoForge Supporter

    yes. SuExec is enabled in all the websites


    the /var/www/html was empty. I have created one:
    Code:
    # touch /var/www/html/index.html
    
    then I have restarted apache but still there is the error.
    Code:
    PHP Fatal error:  xc_fcntl_create: open(/var/www/clients/client2/web4/tmp/.xcache.5007.17157.2.mutex, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
    [Wed May 24 09:16:02.266272 2017] [authz_core:error] [pid 17153] [client 127.0.0.1:52325] AH01630: client denied by server configuration: /var/www/html/
    
    The /html is owned by root:root:
    Code:
    # ls -lha /var/www/
    drwxr-xr-x  2 root  root  4,0K feb 24 19:40 html
    
    every website /tmp is owned by its own web:client:
    Code:
    # ls -lha /var/www/clients/client*/web*/ |grep tmp
    drwxrwx--- 2 web1 client1 4,0K may  8 00:01 tmp
    drwxrwx---  2 5014 client2 4,0K mar 29 23:24 tmp
    drwxrwx---  2 web12 client2 4,0K may 24 00:25 tmp
    drwxrwx---  2 web2 client2 4,0K mar 21  2016 tmp
    drwxrwx---  2 web3 client2 4,0K abr  8 18:39 tmp
    drwxrwx---  2 web4 client2 132K may  8 01:08 tmp
    drwxrwx---  2 web5 client2 4,0K mar 21  2016 tmp
    drwxrwx---  2 web6 client2  76K may 24 08:59 tmp
    drwxrwx---  2 web7 client2 4,0K mar 21  2016 tmp
    drwxrwx---  2 web8 client2 4,0K jul 31  2016 tmp
    drwxrwx---  2 web9 client2 4,0K may  7 00:00 tmp
    

    although there are some differences in the atttributes:
    Code:
    # lsattr /var/www/clients/client*/web*/ | grep tmp
    ----i--------e-- /var/www/clients/client1/web1/tmp
    ----i--------e-- /var/www/clients/client2/web11/tmp
    -------------e-- /var/www/clients/client2/web12/tmp
    ----i--------e-- /var/www/clients/client2/web2/tmp
    ----i--------e-- /var/www/clients/client2/web3/tmp
    ----i-----I--e-- /var/www/clients/client2/web4/tmp
    ----i--------e-- /var/www/clients/client2/web5/tmp
    ----------I--e-- /var/www/clients/client2/web6/tmp
    ----i--------e-- /var/www/clients/client2/web7/tmp
    ----i--------e-- /var/www/clients/client2/web8/tmp
    ----i--------e-- /var/www/clients/client2/web9/tmp
    
    sorry, if the question is about the main /tmp of the server :
    Code:
    drwxrwxrwt  8 root root 4,0K wed 24/05/2017 09:45 tmp/
    
    #cat /etc/fstab
    /var/TMP /tmp ext3 loop,nosuid,noexec,rw 0 0
    
     
    Last edited: May 24, 2017
  14. till

    till Super Moderator Staff Member ISPConfig Developer

    Please try this:

    chattr -i /var/www/clients/*/*/tmp

    I wonder why tmp has the immutable attribute set on your server, as this attribute should only be set on the web root folder e.g. /var/www/clients/client1/web1
     
  15. MaxT

    MaxT Active Member HowtoForge Supporter

    I don't know the cause, probably it was an old attempt to solve a permission problems.
    Now after applying chattr -i /var/www/clients/*/*/tmp I see that still there are 2 with different attributes:

    Code:
    # lsattr /var/www/clients/*/*/ | grep tmp
    -------------e-- /var/www/clients/client2/web12/tmp
    -------------e-- /var/www/clients/client2/web2/tmp
    -------------e-- /var/www/clients/client2/web3/tmp
    ----------I--e-- /var/www/clients/client2/web4/tmp
    -------------e-- /var/www/clients/client2/web5/tmp
    ----------I--e-- /var/www/clients/client2/web6/tmp
    -------------e-- /var/www/clients/client2/web7/tmp
    -------------e-- /var/www/clients/client2/web8/tmp
    -------------e-- /var/www/clients/client2/web9/tmp
    
    that "I" belongs to "Indexed directory" according the lsattr manual.
    It means these two /tmp folders still are immutable?. These two /tmp are the only ones with Wordpress installations.

    I read: https://unix.stackexchange.com/questions/32256/whats-the-meaning-of-output-of-lsattr
    "The 'I' attribute is used by the htree code to indicate that a directory is being indexed using hashed trees. It may not be set or reset using chattr(1), although it can be displayed by lsattr(1)."


    The XCache error has disappeared although the [authz_core:error] still is present:
    Code:
    [Wed May 24 11:02:01.390838 2017] [authz_core:error] [pid 25433] [client 127.0.0.1:53312] AH01630: client denied by server configuration: /var/www/html/
    
    all the info I can find in internet says this is an error because old directives Apache 2.2 "Order allow, deny.." . However, I search inside /etc/apache2 with #grep -ir 'allow,' ./ * and #grep -ir 'deny,' ./ * and all are commented. There are only quite"AllowOverride " but it seems this is nor related. :(
     
  16. MaxT

    MaxT Active Member HowtoForge Supporter

    this [authz_core:error] is specially bad over /var/www/clients/clientX/webX/tmp because the write of sessions. It is solved by changing the /tmp permissions from 0770 to 0777. It works and I cannot find another way to do it.
    According the chattr manual, the 'I' attribute cannot be changed although with apache 2.4 it impedes the php scripts to write sessions. It didn't happen with Apache 2.2

    I ask: Change permissions from 0770 to 0777 in /tmp can be a risk of security?
     
  17. till

    till Super Moderator Staff Member ISPConfig Developer

    Yes, and this change is not needed. The authz_core:error is not about the tmp dirs. The tmp dir error was the one with xcache which is independant from the authz_core:error.

    What you can try is that you add something like this at the end of apache2.conf file:

    <Directory /var/www/html/>
    Require all granted
    </Directory>
     
    MaxT likes this.
  18. MaxT

    MaxT Active Member HowtoForge Supporter

    it worked. Thank you! :)
     

Share This Page