Problem with mod_fcgid

I had a perfectly working server. A couple of sites were using a self-signed SSL certificate which worked as expected - the browser baulked at it but accepting the certificate meant I could test everything under SSL.

I bought a multi-domain Comodo SSL certificate and installed it on the two relevant sites, At that point I started getting a raft of errors:

Code:

[Wed Jan 06 07:39:30 2016] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Wed Jan 06 07:39:30 2016] [warn] RSA server certificate CommonName (CN) `Webmaster' does NOT match server name!?
[Wed Jan 06 07:39:30 2016] [notice] suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec)
[Wed Jan 06 07:39:31 2016] [notice] Digest: generating secret for digest authentication ...
[Wed Jan 06 07:39:31 2016] [notice] Digest: done
[Wed Jan 06 07:39:31 2016] [notice] FastCGI: process manager initialized (pid 22447)
[Wed Jan 06 07:39:31 2016] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Wed Jan 06 07:39:31 2016] [warn] RSA server certificate CommonName (CN) `Webmaster' does NOT match server name!?
[Wed Jan 06 07:39:32 2016] [warn] pid file /var/run/apache2.pid overwritten -- Unclean shutdown of previous Apache run?
[Wed Jan 06 07:39:32 2016] [notice] Apache/2.2.22 (Ubuntu) DAV/2 mod_fastcgi/mod_fastcgi-SNAP-0910052141 mod_fcgid/2.3.6 PHP/5.3.10-1ubuntu3.21 with Suhosin-Patch mod_ssl/2.2.22 OpenSSL/1.0.1 mod_qos/9.76 mod_ruby/1.2.6 Ruby/1.8.7(2011-06-30) mod_wsgi/3.3 Python/2.7.3 configured -- resuming normal operations
[Wed Jan 06 07:40:01 2016] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Wed Jan 06 07:40:01 2016] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Wed Jan 06 07:40:01 2016] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Wed Jan 06 07:40:01 2016] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Wed Jan 06 07:40:01 2016] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Wed Jan 06 07:40:01 2016] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Wed Jan 06 07:40:01 2016] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Wed Jan 06 07:40:01 2016] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Wed Jan 06 07:40:01 2016] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Wed Jan 06 07:40:02 2016] [alert] Child 22452 returned a Fatal error... Apache is exiting!
[Wed Jan 06 07:54:13 2016] [emerg] (22)Invalid argument: couldn't release the accept mutex

Having done some searches for what the cause might be I made several changes (yes, I wish I'd kept a note of all of them :/ ).
Two major changes which allowed the server to work (sort of) were:
(1) Load the right key file for the two SSL sites, one which doesn't have the passphrase in it!
(2) Add "AcceptMutex fcntl" to Apache's config file.
Now what happens is that Apache runs but once/minute this happens:

Code:

[emerg] (22)Invalid argument: mod_fcgid: can't lock process table in PM, pid 1680
[error] mod_fcgid: fcgid process manager died, restarting the server
[notice] SIGHUP received.  Attempting to restart

and another process like this appears:

Code:

  3378     ispconfig     18:06     /usr/bin/php-cgi -d magic_quotes_gpc=off -d session.save_path=/usr/local/ispconf ...
      3380     ispconfig     18:06     /usr/bin/php-cgi -d magic_quotes_gpc=off -d session.save_path=/usr/local/ispconf ...

...and Apache restarts. Using web sites being served, you don't see this happening and as the server is, currently, low traffic, it's not a big deal... apart from, I have to kill a load of processes every now & again.

I have searched for an answer to this problem and done some investigating, so far without success. I'm guessing that, unless you've seen this before, you'd need more information on the environment. This is the basic stuff - let me know what else you might need.
Operating system Ubuntu Linux 12.04.5
Kernel and CPU Linux 3.14.32-xxxx-grs-ipv6-64 on x86_64
Processor Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz, 8 cores
RAM 32 GB
Disc 2 TB, mirrored
Apache version 2.2.22
MySQL version 5.5.46
php5 5.3.10-1ubuntu3.21
libapache2-mod-fcgid 1:2.3.6-1.1ubuntu0.1

ISPConfig Apache VHost:

Code:

  ServerAdmin webmaster@localhost
 
  <FilesMatch "\.ph(p3?|tml)$">
    SetHandler None
  </FilesMatch>
 
  <IfModule mod_fcgid.c>
DocumentRoot /var/www/ispconfig/
    SuexecUserGroup ispconfig ispconfig
    <Directory /var/www/ispconfig/>
      Options -Indexes +FollowSymLinks +MultiViews +ExecCGI
      AllowOverride AuthConfig Indexes Limit Options FileInfo
      <FilesMatch "\.php$">
          SetHandler fcgid-script
      </FilesMatch>
      FCGIWrapper /var/www/php-fcgi-scripts/ispconfig/.php-fcgi-starter .php
            Order allow,deny
      Allow from all
          </Directory>
    IPCCommTimeout  7200
    MaxRequestLen 15728640
  </IfModule>
 
  <IfModule mpm_itk_module>
    DocumentRoot /usr/local/ispconfig/interface/web/
    AssignUserId ispconfig ispconfig
    AddType application/x-httpd-php .php
    <Directory /usr/local/ispconfig/interface/web>
      # php_admin_value open_basedir "/usr/local/ispconfig/interface:/usr/share:/tmp"
      Options +FollowSymLinks
      AllowOverride None
            Order allow,deny
      Allow from all
            php_value magic_quotes_gpc        0
    </Directory>
  </IfModule>
 
  # ErrorLog /var/log/apache2/error.log
  # CustomLog /var/log/apache2/access.log combined
  ServerSignature Off
 
  <IfModule mod_security2.c>
    SecRuleEngine Off
  </IfModule>

  # SSL Configuration
  SSLEngine On
  SSLProtocol All -SSLv2 -SSLv3
  SSLCertificateFile /usr/local/ispconfig/interface/ssl/ispserver.crt
  SSLCertificateKeyFile /usr/local/ispconfig/interface/ssl/ispserver.key
  #SSLCACertificateFile /usr/local/ispconfig/interface/ssl/ispserver.bundle

 

I know - I just made a mistake at first & used the wrong file contents. In fact, it did start but no SSL sites were
accessible as Mod_SSL simply refused to work. Other sites were OK. I was surprised at that - I thought it wouldn't have started :/
Later: I did an Apache Reload - I now know that that's what happens if you only reload. If you stop/start Apache, it won't.

Neither have I! But, according to the posts I found elsewhere it's not that uncommon. I find it very odd that the thing that "broke" Apache was changing from a self-signed CRT to a real one :?

As far as I can see, the website that is causing the problem is the ISPConfig UI - as time goes on, the number of processes like this:

Code:

/usr/bin/php-cgi -d magic_quotes_gpc=off -d session.save_path=/usr/local/ispconfig/interface/temp

increases. What I haven't though to check, up until now, is if a new one occurs *only* if I am using the UI whilst Apache crashes. If I'm wrong, I can't see how to find out which site is causing the problem - the log entries don't identify it.

It is - but the whole point of an LTS release is that all important updates are backported so you don't have to change release every two years. A case of not fixing it if it isn't broken (yes, I know, mine now appears to be :} ) I have to admit, if I could be totally sure that an upgrade wouldn't break my server,I would seriously consider it... but I can't, so the only way I would do it would be to rent a new server, clean install the latest LTS version and migrate domains one at time. However, that said, I'd value your opinion on the subject.

 

Yes and yes. I only had 3 or 4 sites using FCGID and I'd changed them to SuPHP. I have some sites on PHP-FPM but TBH I can't make up my mind which is best for each site. Given their low traffic rate ATM, it probably makes no difference. Anyway... as the FCGID errors persist, it has to be the ISPConfig UI causing the crash - I don't mean it's ISPConfig's fault, I think it more likely to be an FCGID problem (or a configuration error on my part :/).

Damn - as I now don't have any FCGI sites running, I've nothing t copy from! Is there any reason why the ISPConfig UI won't run under SuPHP, as long as I get the directory, Group & User names etc. right?
I've put together an SuPHP vhost config file for ISPCOnfig but it needs the location of a php.ini file. Any ideas, Till?

Code:

<VirtualHost XX.XX.XX.XX:8080>

DocumentRoot /var/www/ispconfig/
ServerAdmin webmaster@localhost

ErrorLog /var/log/ispconfig/httpd/a1shopdirect.com/error.log
Alias /error/ "/var/www/a1shopdirect.com/web/error/"

  <FilesMatch "\.ph(p3?|tml)$">
    SetHandler None
  </FilesMatch>

    <Directory /var/www/ispconfig/>
      Options -Indexes +FollowSymLinks +MultiViews +ExecCGI
      AllowOverride AuthConfig Indexes Limit Options FileInfo
        Order allow,deny
      Allow from all
    </Directory>

# suphp enabled
<Directory /var/www/ispconfig/>
  <IfModule mod_suphp.c>
    suPHP_Engine on
    suPHP_UserGroup ispconfig ispconfig
    suPHP_ConfigPath ?????????????????????????
    <FilesMatch "\.php[345]?$">
      SetHandler x-httpd-suphp
    </FilesMatch>
    suPHP_AddHandler x-httpd-suphp
  </IfModule>
</Directory>

# add support for apache mpm_itk
<IfModule mpm_itk_module>
  DocumentRoot /usr/local/ispconfig/interface/web/
  AssignUserId ispconfig ispconfig
  AddType application/x-httpd-php .php
  <Directory /usr/local/ispconfig/interface/web>
    Options +FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
    php_value magic_quotes_gpc 0
  </Directory>
</IfModule>

<IfModule mod_security2.c>
    SecRuleEngine Off
  </IfModule>

  ServerSignature Off

  # SSL Configuration
  SSLEngine On
  SSLProtocol All -SSLv2 -SSLv3
  SSLCertificateFile /usr/local/ispconfig/interface/ssl/ispserver.crt
  SSLCertificateKeyFile /usr/local/ispconfig/interface/ssl/ispserver.key
  #SSLCACertificateFile /usr/local/ispconfig/interface/ssl/ispserver.bundle
</VirtualHost>

In parallel, I am going to remove the "AcceptMutex fcntl" and get a capture of the original error messages, then try again to see if there is a solution for that. I will also look into updating FCGID as the Ubuntu version is way behind, as is so often the case.

Thanks again for your time & patience

 

Thanks Till. Yes, I finally figured out that this is almost certainly not an ISPC problem but something odd about the configuration of my server.

I have updated mod_fcgi to the latest version from the Apache source and that made no difference. I tried your FCGID param suggestions, both types, both in the VHost definition and in the mod_fcgid.conf file but none of those changes made any difference.

I then had to reboot the machine anyway for an OpenSSL update. On looking at the Apache error log after that, I noticed that the FCGID error messages were missing but the SIGTERM to Apache was still happening once/minute. As soon as I used the ISPC UI, the FCGI message started appearing again, too.

So: I realised that the FCGID errors were not the cause of the problem but a *Symptom*. Too cut a long story short: if I disable the one/minute ISPC CRON job, Apache does not get restarted every minute. I have had a quick glance at the four files server.php, check.php, app.inc.php and config.inc.php and I can't see anything untoward there. I am still getting these messages in the ISPC CRON log, at one minute intervals:

Code:

Fri Jan 8 16:45:02 CET 2016 sh: -c: line 0: unexpected EOF while looking for matching `"'
Fri Jan 8 16:45:02 CET 2016 sh: -c: line 1: syntax error: unexpected end of file
Fri Jan 8 16:50:02 CET 2016 sh: -c: line 0: unexpected EOF while looking for matching `"'
Fri Jan 8 16:50:02 CET 2016 sh: -c: line 1: syntax error: unexpected end of file

and a new one:

Code:

Fri Jan 8 16:41:01 CET 2016 usage: ipcrm [ [-q msqid] [-m shmid] [-s semid]
Fri Jan 8 16:41:01 CET 2016 [-Q msgkey] [-M shmkey] [-S semkey] ... ]

which happens irregularly - I suspect when I have manually stopped Apache to fix something.

I did discover that there is a setting for FCGID syntax 2.0/2.2 which was set to 2.0. As I'm running 2.2, I changed that and, lo, the FCGID errors have stopped! So, a partial result

And now I'm stuck again :/

 

I finally remembered Debug mode. If I set it and run the 1/minute CRON job manually (through Webmin), this is what I get in the ISPC log:

Code:

08.01.2016-22:11 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
08.01.2016-22:11 - DEBUG - Found 2 changes, starting update process.
08.01.2016-22:11 - DEBUG - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
08.01.2016-22:11 - DEBUG - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
08.01.2016-22:11 - DEBUG - Network configuration disabled in server settings.
08.01.2016-22:11 - DEBUG - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'.
08.01.2016-22:11 - DEBUG - Processed datalog_id 2294
08.01.2016-22:11 - DEBUG - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
08.01.2016-22:11 - DEBUG - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
08.01.2016-22:11 - DEBUG - Network configuration disabled in server settings.
08.01.2016-22:11 - DEBUG - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'.
08.01.2016-22:11 - DEBUG - Processed datalog_id 2295
08.01.2016-22:11 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
08.01.2016-22:11 - DEBUG - Restarting httpd: service apache2 restart
08.01.2016-22:11 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock

There are 6 web_domain updates in the jobqueue but they are for another server and they are not being processed because the other server is not currently available.

The reason Network Settings is disabled is because I administer that in Webmin (been doing that longer than I've been using ISPC & just never changed).

 

Sorry to add to this... I've just discovered that most web processes are running under www-data and not their "own" group/userid. Apart from the security aspect, this has broken some websites. This happens if suexec is set or not in ISPC for the site. I *think* this happened since I updated mod_fcgid but it might be since I changed the syntax setting in ISPC from 2.0 to 2.2. Looking at a VHost file, there's some extra lines that weren't there last time I looked... and some I would not expect to see if the site were running under PHP-FPM. I did wonder if it was changing the sites to On Demand for PHP-FPM, so I tried changing one back to Dynamic but that had no effect.

I won't be around most of tomorrow to take any action wit the web server, so if you can spare some time to think about this lot, no rush :}

I'm beginning to think there's something seriously broken on this server

 

Sorry Till, I know that but wasn't being very clear. I shouldn't have tried fixing that at 1 in the morning :/
Sites which *should* be running under php-fpm and should therefore be running under clientxx/webyy (which is now almost all of them) are not - they're running under apache (I presume) and www-data.

I cannot find anything in the logs which would indicate why this is happening. I have restarted php-fpm then restarted Apache after doing a web site resync in ISPC.

Is there any information I can post that might make it easier to see what is going on?

I have been using a similar setup for several years now and hadn't really gone into things very deeply - they just worked. I think it's now about time I learnt a lot more :/

 

Hi Till - I wasn't expecting a reply on a Sunday

.
I understand that perfectly - that was my attempt to explain what was happening. I'll try again....

The sites were created normally in ISPConfig. Originally, some were running under php-fpm, some under fcgi and some under suphp. There were historical reasons for that, as well as me not doing enough research into the best way to run them and keeping up with developments.

When you said about php-fpm being the best way to run sites generally (at least, that's what I understood), I changed virtually all of them to php-fpm. Once all the updates had been applied, I checked and virtually all of them were running under Apache as www. The only one running under php-fpm was one which was not enabled in ISPC!

That's what used to happen but it isn't now. If I create a new site and set it to run under php-fpm, it runs under Apache instead. Just in case, I tried it both with suexec on and off. Just FYI, the server setup isn't totally as per the Perfect setup - I don't use either of the two firewalls recommended, nor fail2ban: I use CSF.

I have overcome the problem temporarily by reverting to fcgid and suexec. Performance is not an issue ATM, so I can now resolve the real problem without rushing and I have an operational server. BTW, I'm pretty sure this problem originated with one of two events:
Either
1) Changing the fcgi syntax in ISPC settings from 2.0 to 2.2
or
2) updating mod-fcgid manually (downloading the source/config/make/install)
I did a web site resync in ISPC but that hasn't helped. Oh, and the ISPC job queue is now empty.

One other thing 9and this may be a separate problem but it's what started this off): the 1/minute ISPC Cron job is still restarting Apache every time it runs, regardless of whether there is anything in the job queue. For the moment, I have that CRON job de-activated and I run it manually if I make any change in ISPC.

I did write a bit of php myself to check what was going on with a couple of the sites and it confirmed that they were running under www-data despite being configured to run under php-fpm. I can actually see this easily anyway by using the process display in Webmin but I wanted to be sure.

I have a feeling that I've done something that has broken the server in a way I probably can't fix, so I will probably have to rebuild it in the near future, although if you have any suggestions that might help me fix the problems, I'd be very grateful.

Thanks again for all your help & patience.