Apache restarts 200 times every week

Hi all,

After I upgraded the ISPConfig to 3.2.9 version, Apache restarts over 200 times for almost an hour every Saturday (around 00:05), it causes all websites unstable during that hour every week, any hints & help?

After debug was enabled, there are 209 records during 1674230837-1674230838 (January 21, 2023 12:07:17) in sys_datalog, and in the data field the difference between old and new is only last_quota_notification:
< ["last_quota_notification"]=> string(10) "2023-01-21"
> ["last_quota_notification"]=> string(10) "2023-01-14"

/var/log/ispconfig/cron.log shown as below:

Code:

Sat Jan 21 00:08:01 HKT 2023 21.01.2023-00:08 - DEBUG [server:177] - Found 209 changes, starting update process.
Sat Jan 21 00:08:01 HKT 2023 21.01.2023-00:08 - DEBUG [plugins.inc:118] - Calling function 'ssl' from plugin 'apache2_plugin' raised by event 'web_domain_update'.
Sat Jan 21 00:08:01 HKT 2023 21.01.2023-00:08 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'apache2_plugin' raised by event 'web_domain_update'.
Sat Jan 21 00:08:01 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: chattr -i '/var/www/clients/client0/web4' - return code: 0
Sat Jan 21 00:08:01 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: chattr +i '/var/www/clients/client0/web4' - return code: 0
Sat Jan 21 00:08:01 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: df -T '/var/www/clients/client0/web4'|awk 'END{print $2,$NF}' - return code: 0
Sat Jan 21 00:08:01 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: which 'setquota' 2> /dev/null - return code: 0
Sat Jan 21 00:08:01 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: setquota -u 'web4' '12533760' '12534784' 0 0 -a &> /dev/null - return code: 0
Sat Jan 21 00:08:01 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: setquota -T -u 'web4' 604800 604800 -a &> /dev/null - return code: 0
Sat Jan 21 00:08:01 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: chattr +i '/var/www/clients/client0/web4' - return code: 0
Sat Jan 21 00:08:01 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 1
Sat Jan 21 00:08:01 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apachectl' 2> /dev/null - return code: 0
Sat Jan 21 00:08:01 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 1
Sat Jan 21 00:08:01 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apachectl' 2> /dev/null - return code: 0
Sat Jan 21 00:08:01 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: chattr -i '/var/www/php-fcgi-scripts/web4/.php-fcgi-starter' - return code: 0
Sat Jan 21 00:08:01 HKT 2023 21.01.2023-00:08 - DEBUG [apache2 plugin.inc:1602] - Creating fastcgi starter script: /var/www/php-fcgi-scripts/web4/.php-fcgi-starter
Sat Jan 21 00:08:01 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: chattr +i '/var/www/php-fcgi-scripts/web4/.php-fcgi-starter' - return code: 0
Sat Jan 21 00:08:01 HKT 2023 21.01.2023-00:08 - DEBUG [apache2 plugin.inc:1874] - Writing the vhost file: /etc/httpd/conf/sites-available/example1.com.vhost
Sat Jan 21 00:08:01 HKT 2023 21.01.2023-00:08 - DEBUG [apache2 plugin.inc:1992] - Apache status is: running
Sat Jan 21 00:08:01 HKT 2023 21.01.2023-00:08 - DEBUG [services.inc:56] - Calling function 'restartHttpd' from module 'web_module'.
Sat Jan 21 00:08:01 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2082] - Trying to use Systemd to restart service
Sat Jan 21 00:08:02 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: systemctl is-enabled 'httpd' 2>&1 - return code: 0
Sat Jan 21 00:08:02 HKT 2023 21.01.2023-00:08 - DEBUG [web module.inc:246] - Restarting httpd: systemctl restart httpd.service
Sat Jan 21 00:08:13 HKT 2023 21.01.2023-00:08 - DEBUG [apache2 plugin.inc:1995] - Apache restart return value is: 0
Sat Jan 21 00:08:15 HKT 2023 21.01.2023-00:08 - DEBUG [apache2 plugin.inc:2006] - Apache online status after restart is: running
Sat Jan 21 00:08:15 HKT 2023 21.01.2023-00:08 - DEBUG [modules.inc:240] - Processed datalog_id 5485
Sat Jan 21 00:08:15 HKT 2023 21.01.2023-00:08 - DEBUG [plugins.inc:118] - Calling function 'ssl' from plugin 'apache2_plugin' raised by event 'web_domain_update'.
Sat Jan 21 00:08:15 HKT 2023 21.01.2023-00:08 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'apache2_plugin' raised by event 'web_domain_update'.
Sat Jan 21 00:08:15 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: chattr -i '/var/www/clients/client0/web5' - return code: 0
Sat Jan 21 00:08:15 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: chattr +i '/var/www/clients/client0/web5' - return code: 0
Sat Jan 21 00:08:15 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: df -T '/var/www/clients/client0/web5'|awk 'END{print $2,$NF}' - return code: 0
Sat Jan 21 00:08:15 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: which 'setquota' 2> /dev/null - return code: 0
Sat Jan 21 00:08:15 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: setquota -u 'web5' '5242880' '5243904' 0 0 -a &> /dev/null - return code: 0
Sat Jan 21 00:08:15 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: setquota -T -u 'web5' 604800 604800 -a &> /dev/null - return code: 0
Sat Jan 21 00:08:15 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: chattr +i '/var/www/clients/client0/web5' - return code: 0
Sat Jan 21 00:08:15 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 1
Sat Jan 21 00:08:15 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apachectl' 2> /dev/null - return code: 0
Sat Jan 21 00:08:15 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 1
Sat Jan 21 00:08:15 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apachectl' 2> /dev/null - return code: 0
Sat Jan 21 00:08:16 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: chattr -i '/var/www/php-fcgi-scripts/web5/.php-fcgi-starter' - return code: 0
Sat Jan 21 00:08:16 HKT 2023 21.01.2023-00:08 - DEBUG [apache2 plugin.inc:1602] - Creating fastcgi starter script: /var/www/php-fcgi-scripts/web5/.php-fcgi-starter
Sat Jan 21 00:08:16 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: chattr +i '/var/www/php-fcgi-scripts/web5/.php-fcgi-starter' - return code: 0
Sat Jan 21 00:08:16 HKT 2023 21.01.2023-00:08 - DEBUG [apache2 plugin.inc:1874] - Writing the vhost file: /etc/httpd/conf/sites-available/example2.com.vhost
Sat Jan 21 00:08:16 HKT 2023 21.01.2023-00:08 - DEBUG [apache2 plugin.inc:1992] - Apache status is: running
Sat Jan 21 00:08:16 HKT 2023 21.01.2023-00:08 - DEBUG [services.inc:56] - Calling function 'restartHttpd' from module 'web_module'.
Sat Jan 21 00:08:16 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2082] - Trying to use Systemd to restart service
Sat Jan 21 00:08:16 HKT 2023 21.01.2023-00:08 - DEBUG [system.inc:2399] - safe_exec cmd: systemctl is-enabled 'httpd' 2>&1 - return code: 0
Sat Jan 21 00:08:16 HKT 2023 21.01.2023-00:08 - DEBUG [web module.inc:246] - Restarting httpd: systemctl restart httpd.service
Sat Jan 21 00:08:22 HKT 2023 21.01.2023-00:08 - DEBUG [apache2 plugin.inc:1995] - Apache restart return value is: 0
Sat Jan 21 00:08:24 HKT 2023 21.01.2023-00:08 - DEBUG [apache2 plugin.inc:2006] - Apache online status after restart is: running
Sat Jan 21 00:08:24 HKT 2023 21.01.2023-00:08 - DEBUG [modules.inc:240] - Processed datalog_id 5486

 

Hi Till,
Thank you for adding issue report for this.

For Resync question, yes, I am sure I did nothing because it is at midnight every 7 days.

 

Hi Till,
May I know if there is any fix for this?

I woke up and found the web server had been down for whole night, so I check the log again.

Last midnight (Satuday 00:07:01) the huge restart problem happened again.

Code:

Sat Jan 28 00:07:01 HKT 2023 28.01.2023-00:07 - DEBUG [server:177] - Found 209 changes, starting update process.

I also found Apache can't be restarted in /var/log/ispconfig/cron.log

Code:

Sat Jan 28 00:11:37 HKT 2023 28.01.2023-00:11 - WARNING - Apache did not restart after the configuration change for website example.com. Reverting the configuration. Saved non-working config as /etc/httpd/conf/sites-available/example.com.vhost.err

There is no difference between example.com.vhost & example.com.vhost.err, and then i found below in /var/log/httpd/error_log

Code:

[Sat Jan 28 00:11:30.751027 2023] [core:emerg] [pid 1638] (28)No space left on device: AH00023: Couldn't create the ssl-cache mutex
AH00016: Configuration Failed

After server was rebooted, Apache is up.

Seems the problem goes worse, last week Apache restarted many times for an hour, but eventually was up. Last night it cannot be up until my reboot.

 

Code:

No space left on device: AH00023

What is it that triggers this behaviour? Is the disk full? Is one website over quota? Is it just random?

 

No, it is not disk full. The problem is that Apache restarted many times and cause reaching semaphore limit:
https://serverfault.com/questions/9...3-couldnt-create-the-mpm-accept-mutex-when-re

 

Thank You Till, I have removed the file and I will track that issue status in git, thank you very much.

 

I've created a fix for this in https://git.ispconfig.org/ispconfig/ispconfig3/-/merge_requests/1734
Could you verify that is solves your problem?