My box continually locks up with loads of over 190. In the Apache error log, I see this for several hours prior to the lockup: Code: [Mon Apr 11 13:43:28 2011] [warn] mod_fcgid: process 12274 graceful kill fail, sending SIGKILL [Mon Apr 11 13:43:34 2011] [warn] mod_fcgid: process 12304 graceful kill fail, sending SIGKILL [Mon Apr 11 13:43:34 2011] [warn] mod_fcgid: process 12297 graceful kill fail, sending SIGKILL [Mon Apr 11 13:43:34 2011] [warn] mod_fcgid: process 12295 graceful kill fail, sending SIGKILL [Mon Apr 11 13:43:34 2011] [warn] mod_fcgid: process 12299 graceful kill fail, sending SIGKILL [Mon Apr 11 13:43:34 2011] [warn] mod_fcgid: process 12296 graceful kill fail, sending SIGKILL [Mon Apr 11 13:43:34 2011] [warn] mod_fcgid: process 12274 graceful kill fail, sending SIGKILL [Mon Apr 11 13:43:40 2011] [warn] mod_fcgid: process 12304 graceful kill fail, sending SIGKILL [Mon Apr 11 13:43:40 2011] [warn] mod_fcgid: process 12297 graceful kill fail, sending SIGKILL [Mon Apr 11 13:43:40 2011] [warn] mod_fcgid: process 12295 graceful kill fail, sending SIGKILL [Mon Apr 11 13:43:40 2011] [warn] mod_fcgid: process 12299 graceful kill fail, sending SIGKILL [Mon Apr 11 13:43:40 2011] [warn] mod_fcgid: process 12296 graceful kill fail, sending SIGKILL [Mon Apr 11 13:43:40 2011] [warn] mod_fcgid: process 12274 graceful kill fail, sending SIGKILL [Mon Apr 11 13:43:46 2011] [warn] mod_fcgid: process 12304 graceful kill fail, sending SIGKILL [Mon Apr 11 13:43:46 2011] [warn] mod_fcgid: process 12297 graceful kill fail, sending SIGKILL [Mon Apr 11 13:43:46 2011] [warn] mod_fcgid: process 12295 graceful kill fail, sending SIGKILL [Mon Apr 11 13:43:46 2011] [warn] mod_fcgid: process 12299 graceful kill fail, sending SIGKILL [Mon Apr 11 13:43:46 2011] [warn] mod_fcgid: process 12296 graceful kill fail, sending SIGKILL [Mon Apr 11 13:43:46 2011] [warn] mod_fcgid: process 12274 graceful kill fail, sending SIGKILL [Mon Apr 11 13:43:52 2011] [warn] mod_fcgid: process 12304 graceful kill fail, sending SIGKILL [Mon Apr 11 13:43:52 2011] [warn] mod_fcgid: process 12297 graceful kill fail, sending SIGKILL [Mon Apr 11 13:43:52 2011] [warn] mod_fcgid: process 12295 graceful kill fail, sending SIGKILL ..... [Mon Apr 11 13:53:51 2011] [warn] mod_fcgid: process 13068 graceful kill fail, sending SIGKILL [Mon Apr 11 13:53:51 2011] [warn] mod_fcgid: process 13066 graceful kill fail, sending SIGKILL [Mon Apr 11 13:54:44 2011] [warn] mod_fcgid: process 13081 graceful kill fail, sending SIGKILL [Mon Apr 11 13:55:06 2011] [warn] mod_fcgid: process 13109 graceful kill fail, sending SIGKILL [Mon Apr 11 13:55:20 2011] [warn] mod_fcgid: process 13109 graceful kill fail, sending SIGKILL [Mon Apr 11 13:55:26 2011] [warn] mod_fcgid: process 13112 graceful kill fail, sending SIGKILL [Mon Apr 11 13:55:26 2011] [warn] mod_fcgid: process 13109 graceful kill fail, sending SIGKILL In the end, I have to have the ISP hard-reboot the server so that it will be back online and accessible. Any ideas? This has been destroying my productivity since switching to ISPC3 last year...
Therae are no known problems with ispconfig and fcgid in the current versions. Which ISPConfig version do you use, what are the hardware specifications of the server etc. And please post the fcgi starter file of a affected website.
I'm running Debian Squeeze x64 (happened on Lenny x64 also) Linux 2.6.32-5-amd64 on x86_64. It's a Intel(R) Pentium(R) D CPU 3.00GHz 2 cores, 4 GB RAM, and 2x 300GB SATA2 hdds. I've got ISPC 3.0.3.2 installed. Here are several FCGI starter scripts from almost all of the sites that end up with the crazy spawning of processes and lag crashing. Code: cat /var/www/php-fcgi-scripts/web11/.php-fcgi-starter #!/bin/sh PHPRC="/var/www/conf/web11" export PHPRC PHP_DOCUMENT_ROOT="/var/www/clients/client1/web11" export PHP_DOCUMENT_ROOT # The variable PHP_FCGI_CHILDREN is onyl useful for lighty or nginx as apache # mod_fcgi will control the number of childs themself and never use the additional processes. # PHP_FCGI_CHILDREN=1 # export PHP_FCGI_CHILDREN PHP_FCGI_MAX_REQUESTS=5000 export PHP_FCGI_MAX_REQUESTS exec /usr/bin/php-cgi \ -d open_basedir="/var/www/clients/client1/web11/web:/var/www/clients/client1/web11/tmp:/var/www/derekgordon.com/web:/srv/www/derekgordon.com/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin" \ -d upload_tmp_dir=/var/www/clients/client1/web11/tmp \ -d session.save_path=/var/www/clients/client1/web11/tmp \ $1 Code: my:/home/dingle# cat /var/www/php-fcgi-scripts/web5/.php-fcgi-starter #!/bin/sh PHPRC="/var/www/conf/web5" export PHPRC PHP_DOCUMENT_ROOT="/var/www/clients/client1/web5" export PHP_DOCUMENT_ROOT # The variable PHP_FCGI_CHILDREN is onyl useful for lighty or nginx as apache # mod_fcgi will control the number of childs themself and never use the additional processes. # PHP_FCGI_CHILDREN=1 # export PHP_FCGI_CHILDREN PHP_FCGI_MAX_REQUESTS=5000 export PHP_FCGI_MAX_REQUESTS exec /usr/bin/php-cgi \ -d open_basedir="/var/www/clients/client1/web5/web:/var/www/clients/client1/web5/tmp:/var/www/areyouliberal.com/web:/srv/www/areyouliberal.com/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin" \ -d upload_tmp_dir=/var/www/clients/client1/web5/tmp \ -d session.save_path=/var/www/clients/client1/web5/tmp \ export LD_PRELOAD = amon.so $1 Code: my:/home/dingle# cat /var/www/php-fcgi-scripts/web8/.php-fcgi-starter #!/bin/sh PHPRC="/var/www/conf/web8" export PHPRC PHP_DOCUMENT_ROOT="/var/www/clients/client1/web8" export PHP_DOCUMENT_ROOT # The variable PHP_FCGI_CHILDREN is onyl useful for lighty or nginx as apache # mod_fcgi will control the number of childs themself and never use the additional processes. # PHP_FCGI_CHILDREN=1 # export PHP_FCGI_CHILDREN PHP_FCGI_MAX_REQUESTS=5000 export PHP_FCGI_MAX_REQUESTS exec /usr/bin/php-cgi \ -d open_basedir="/var/www/clients/client1/web8/web:/var/www/clients/client1/web8/tmp:/var/www/inter5.org/web:/srv/www/inter5.org/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin" \ -d upload_tmp_dir=/var/www/clients/client1/web8/tmp \ -d session.save_path=/var/www/clients/client1/web8/tmp \ export LD_PRELOAD = amon.so $1 Code: my:/home/dingle# cat /var/www/php-fcgi-scripts/web35/.php-fcgi-starter #!/bin/sh PHPRC="/var/www/conf/web35" export PHPRC PHP_DOCUMENT_ROOT="/var/www/clients/client1/web35" export PHP_DOCUMENT_ROOT # The variable PHP_FCGI_CHILDREN is onyl useful for lighty or nginx as apache # mod_fcgi will control the number of childs themself and never use the additional processes. # PHP_FCGI_CHILDREN=1 # export PHP_FCGI_CHILDREN PHP_FCGI_MAX_REQUESTS=5000 export PHP_FCGI_MAX_REQUESTS exec /usr/bin/php-cgi \ -d open_basedir="/var/www/clients/client1/web35/web:/var/www/clients/client1/web35/tmp:/var/www/masonicleaks.com/web:/srv/www/masonicleaks.com/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin" \ -d upload_tmp_dir=/var/www/clients/client1/web35/tmp \ -d session.save_path=/var/www/clients/client1/web35/tmp \ $1my:/home/dingle#
Does these lines are really like this: -d session.save_path=/var/www/clients/client1/web5/tmp \ export LD_PRELOAD = amon.so $1 it has to be: -d session.save_path=/var/www/clients/client1/web5/tmp \ $1 The other parts of the file look fine. You should check your apache configuration file and limit the number of apache instances that get spawned.
I was playing around with amon earlier. So, those entries weren't always there. When you say the max instances, are you talking about the FCGI part in the vhosts files? If so, what would you recommend the settings to be?
I'am talking about the mpm settings in the apache2.conf file. But you can take a look in the vhost file as well, be default the values in the vhost file should be: # IdleTimeout n (3600 seconds) # An idle fastcgi application will be terminated after IdleTimeout seconds. IdleTimeout 3600 # ProcessLifeTime n (7200 seconds) # A fastcgi application will be terminated if lifetime expired, even no error is detected. ProcessLifeTime 7200 # MaxProcessCount n (1000) # The max count of total fastcgi process count. # MaxProcessCount 1000 # DefaultMinClassProcessCount n (3) # The minimum number of fastcgi application instances for any one fastcgi application. # Idle fastcgi will not be killed if their count is less than n # Set this to 0, and tweak IdleTimeout DefaultMinClassProcessCount 0 # DefaultMaxClassProcessCount n (100) # The maximum number of fastcgi application instances allowed to run for # particular one fastcgi application. DefaultMaxClassProcessCount 100 # IPCConnectTimeout n (3 seconds) # The connect timeout to a fastcgi application. IPCConnectTimeout 8 # IPCCommTimeout n (20 seconds) # The communication timeout to a fastcgi application. Please increase this # value if your CGI have a slow initialization or slow respond. IPCCommTimeout 360 # BusyTimeout n (300 seconds) # A fastcgi application will be terminated if handing a single request # longer than busy timeout. BusyTimeout 300 you can e.g. tune the MaxProcessCount and DefaultMaxClassProcessCount parameters.
Right, and in apache2 I see the following with mpm involved: Code: IfModule mpm_prefork_module> StartServers 5 MinSpareServers 5 MaxSpareServers 10 MaxClients 150 MaxRequestsPerChild 0 </IfModule> Code: <IfModule mpm_worker_module> StartServers 2 MaxClients 150 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 0 </IfModule> what would you recommend for changing? Would max clients of 50 be too low? With a MaxSpareThreads of 45?
