Apache won't run after ISPConfig upgrade

Discussion in 'ISPConfig 3 Priority Support' started by Nilpo, Nov 18, 2014.

  1. Nilpo

    Nilpo Member HowtoForge Supporter

    I just upgraded ISPConfig from version 3.0.5.4p1 to 3.0.5.4p5 using the upgrade script, accepting all of the default options. Now Apache will not run with the error that httpd is dead but the pid file still exists. Here's my Apache error log.

    Code:
    [Tue Nov 18 02:15:22 2014] [notice] Digest: done
    [Tue Nov 18 02:15:23 2014] [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
    [Tue Nov 18 02:15:23 2014] [notice] mod_python: Creating 4 session mutexes based on 10 max processes and 0 max threads.
    [Tue Nov 18 02:15:23 2014] [notice] mod_python: using mutex_directory /tmp 
    [Tue Nov 18 02:15:23 2014] [warn] [mod_spdy/0.9.4.2-413] [5064:5064:WARNING:mod_spdy.cc(166)] mod_spdy is installed, but has not been enabled in the Apache config. SPDY will not be used by this server.  See http://code.google.com/p/mod-spdy/wiki/ConfigOptions for how to enable.
    [Tue Nov 18 02:15:23 2014] [notice] Apache/2.2.15 (Unix) DAV/2 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 mod_python/3.3.1 Python/2.6.6 mod_ruby/1.3.0 Ruby/1.8.7(2011-06-30) configured -- resuming normal operations
    [Tue Nov 18 02:16:01 2014] [emerg] (43)Identifier removed: couldn't grab the accept mutex
    [Tue Nov 18 02:16:01 2014] [emerg] (43)Identifier removed: couldn't grab the accept mutex
    [Tue Nov 18 02:16:02 2014] [alert] Child 5068 returned a Fatal error... Apache is exiting!
    [Tue Nov 18 02:16:13 2014] [emerg] (22)Invalid argument: couldn't release the accept mutex
    
     
    Last edited: Nov 18, 2014
  2. Nilpo

    Nilpo Member HowtoForge Supporter

    I was able to get around the issue but adding this to my http.conf file.

    Code:
    AcceptMutex fcntl
    But the question still remains, why is this happening after running the ISPC update and how do I fix this properly?
     
  3. till

    till Super Moderator Staff Member ISPConfig Developer

    I did not had that problem yet and it might be that the issue was triggered by the apache restart that the update did and not by the update itself (as it does not change the apache config except of the ispconfig vhost). But as far as I can see, the solution you aplied is the corect one for that error.
     
  4. Nilpo

    Nilpo Member HowtoForge Supporter

    Very good. Thanks for the reply!
     
  5. Nilpo

    Nilpo Member HowtoForge Supporter

    Ok, there are a whole lot of other problems too. I just noticed that all mail is now being deferred. I can't get any of it to go through.

    Also, I'm being blasted by thousands of emails saying "WARNING - Clearing semaphores table for user apache."

    I think Apache is crashing and reloading on every page request.

    I'm using the Perfect server setup on CentOS 6.5.
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    Did you try a server reboot?
     
  7. Nilpo

    Nilpo Member HowtoForge Supporter

    Yes I did. That did not help at all.
     
  8. Nilpo

    Nilpo Member HowtoForge Supporter

    I used the backup option to /var/backup. Is it possible to reverse the upgrade and restore my last configuration from that?
     
  9. till

    till Super Moderator Staff Member ISPConfig Developer

    Sure, thats possible, but I dont think that the ispconfig update is the reason for that problem. To restore it, unpack the backup which is in a subfolder of /var/backup/ and move the files back to their original locations. The backup contains a tar.gz file for /etc, one for /usr/local/ispconfig and a sql dump for the dbispconfig database which can be restored with phpmyadmin.
     
  10. Nilpo

    Nilpo Member HowtoForge Supporter

    Apache seems to have start working again through no effort of my own. I left it alone overnight.

    Postfix still is not delivering any mails. I believe amavis may be to blame. amavisd is listening on port 10024, but I am not able to telnet on that port. I get "connect failed".

    Restarting related services has no effect.

    Also, I would really like to disable the httpd rescue emails. I am getting "httpd is down" emails by the thousands. It makes it very hard to determine what the mail problem is when I have so many of them backlogged.
     
  11. Nilpo

    Nilpo Member HowtoForge Supporter

    Well, I've narrowed down the mail issue to amavis. When I comment out the following line in postfix's main.cf, everything begins working again.

    Code:
    content_filter = amavis:[127.0.0.1]:10024
    The problem is that *should* work. Nothing about my configuration has changed. I have verified that amavis is in fact listening on port 10024 and that it is on the 127.0.0.1 interface. I'm also able to telnet on that port and interface and I do get the amavis-new greeting. I just don't know what else to try.

    I was also able to stop the rescue emails by editing the lib file and having it check the correct interface. My server does not have Apache listening on "localhost".
     
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    Is this a real server or virtual server? Did you try to reinstal amavis?
     
  13. Nilpo

    Nilpo Member HowtoForge Supporter

    This is a VPS.

    I have not tried reinstalling amavis. I was working fine and I can't find where anything changed. Very strange. I've even scoured the error logs with no luck. I'll try reinstalling it.
     
  14. till

    till Super Moderator Staff Member ISPConfig Developer

    The amavisd config has not been changed for several versions, so that it has not changed is the result that I have exected. So the ispconfig update has likely only been the trigger to show the issues due to the restart of services and not the cause. My guess is that a limit of your vps has been reached as these rather unspecifoc symptoms in many services are typical for such a limit hit in a vps. Please post the output of:

    cat /proc/user_bancounters
     
  15. Nilpo

    Nilpo Member HowtoForge Supporter

    No such file or directory.
     
  16. Nilpo

    Nilpo Member HowtoForge Supporter

    Sorry, I just caught your typo.

    Code:
    Version: 2.5
           uid  resource                     held              maxheld              barrier                limit              failcnt
          474:  kmemsize                166178487            548478976           1073741824           1073741824                    0
                lockedpages                     0                   11               262144               262144                    0
                privvmpages                829569              3369390  9223372036854775807  9223372036854775807                    0
                shmpages                    14213                16165  9223372036854775807  9223372036854775807                    0
                dummy                           0                    0  9223372036854775807  9223372036854775807                    0
                numproc                       182                  222  9223372036854775807  9223372036854775807                    0
                physpages                  516629               524599                    0               524288                  101
                vmguarpages                     0                    0  9223372036854775807  9223372036854775807                    0
                oomguarpages               431913               538597  9223372036854775807  9223372036854775807                   33
                numtcpsock                     64                  146  9223372036854775807  9223372036854775807                    0
                numflock                       17                   48  9223372036854775807  9223372036854775807                    0
                numpty                          1                    2  9223372036854775807  9223372036854775807                    0
                numsiginfo                      0                   33  9223372036854775807  9223372036854775807                    0
                tcpsndbuf                 1273648              8635904  9223372036854775807  9223372036854775807                    0
                tcprcvbuf                 1048576              2392064  9223372036854775807  9223372036854775807                    0
                othersockbuf               331608              1040672  9223372036854775807  9223372036854775807                    0
                dgramrcvbuf                     0                13080  9223372036854775807  9223372036854775807                    0
                numothersock                  241                  307  9223372036854775807  9223372036854775807                    0
                dcachesize              107296346            498910936            536870912            536870912                    0
                numfile                      5099                 5755  9223372036854775807  9223372036854775807                    0
                dummy                           0                    0  9223372036854775807  9223372036854775807                    0
                dummy                           0                    0  9223372036854775807  9223372036854775807                    0
                dummy                           0                    0  9223372036854775807  9223372036854775807                    0
                numiptent                      37                   40  9223372036854775807  9223372036854775807                    0
    
     
  17. till

    till Super Moderator Staff Member ISPConfig Developer

    Yes, sorry for the typo :)

    As you can see in the last column, your server hit 2 limits in the vm. I guess the pyspages is the likely candidate. Here a short explanation what physpages is:

    https://openvz.org/VSwap

    Is this your own server or did you rent the vm so you dont have access to the virtual machine parameters?
     
  18. Nilpo

    Nilpo Member HowtoForge Supporter

    Unfortunately, I rent this one.
     

Share This Page