I keep getting segmentation faults in the ISPConfig Cron Log. I had the data center check the server and this was the response. Here are the two lines that keep showing up in the ISPConfig Cron Log: Code: Thu Jul 13 09:41:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 6995 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php Thu Jul 13 09:41:01 PDT 2017 Thu Jul 13 09:41:01 PDT 2017 Thu Jul 13 09:41:01 PDT 2017 finished. Thu Jul 13 09:42:02 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 7042 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php Thu Jul 13 09:42:02 PDT 2017 Thu Jul 13 09:42:02 PDT 2017 Thu Jul 13 09:42:02 PDT 2017 finished. The OS is Debian Jessie in an OpenVZ container on dedicated server I rent. ISPConfig 3 version 3.1.5. No pending updates/upgrades available. No fails in the openvz beancounter. Code: Version: 2.5 uid resource held maxheld barrier limit failcnt 30101: kmemsize 205803536 224780288 9223372036854775807 9223372036854775807 0 lockedpages 0 0 9223372036854775807 9223372036854775807 0 privvmpages 2544266 2669855 9223372036854775807 9223372036854775807 0 shmpages 111906 112578 9223372036854775807 9223372036854775807 0 dummy 0 0 0 0 0 numproc 332 381 9223372036854775807 9223372036854775807 0 physpages 2054561 2059473 0 6291456 0 vmguarpages 0 0 9223372036854775807 9223372036854775807 0 oomguarpages 1482392 1482392 9223372036854775807 9223372036854775807 0 numtcpsock 85 128 1801439850948198 1801439850948198 0 numflock 424 430 9223372036854775807 9223372036854775807 0 numpty 2 13 9223372036854775807 9223372036854775807 0 numsiginfo 0 45 9223372036854775807 9223372036854775807 0 tcpsndbuf 1563560 2534848 4611686018427387903 9223372036854775807 0 tcprcvbuf 1494840 2267728 4611686018427387903 9223372036854775807 0 othersockbuf 899368 2370328 4611686018427387903 9223372036854775807 0 dgramrcvbuf 0 27744 9223372036854775807 9223372036854775807 0 numothersock 421 447 1801439850948198 1801439850948198 0 dcachesize 77102998 77231262 9223372036854775807 9223372036854775807 0 numfile 4007 4270 9223372036854775807 9223372036854775807 0 dummy 0 0 0 0 0 dummy 0 0 0 0 0 dummy 0 0 0 0 0 numiptent 150 150 9223372036854775807 9223372036854775807 0 Memory not exhausted Code: # free -m total used free shared buffers cached Mem: 24576 8122 16453 437 0 2107 -/+ buffers/cache: 6014 18561 Swap: 12288 0 12288
Please try to turn on debug loglevel in ISPConfig so we might be able to see which program is failing here. The problem can be related to the hardware like flaky RAM or a mainboard issues as well, or it is an issue in the PHP interpreter itself or third party program which gets called by ISPConfig, but that's less likely.
Here is the output of a section with a segmentation fault (11:09:01) in the ISPConfig Cron Log with debug logging enabled. I don't believe there to be any helpful information in the log for this particular situation. What do you recommend I try next? Code: Fri Jul 14 11:06:01 PDT 2017 Fri Jul 14 11:06:01 PDT 2017 Fri Jul 14 11:06:01 PDT 2017 14.07.2017-11:06 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'. Fri Jul 14 11:06:01 PDT 2017 14.07.2017-11:06 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock Fri Jul 14 11:06:01 PDT 2017 finished. Fri Jul 14 11:07:01 PDT 2017 Fri Jul 14 11:07:01 PDT 2017 Fri Jul 14 11:07:01 PDT 2017 14.07.2017-11:07 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'. Fri Jul 14 11:07:01 PDT 2017 14.07.2017-11:07 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock Fri Jul 14 11:07:01 PDT 2017 finished. Fri Jul 14 11:08:01 PDT 2017 Fri Jul 14 11:08:01 PDT 2017 Fri Jul 14 11:08:01 PDT 2017 14.07.2017-11:08 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'. Fri Jul 14 11:08:01 PDT 2017 14.07.2017-11:08 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock Fri Jul 14 11:08:01 PDT 2017 finished. Fri Jul 14 11:09:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 10114 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php Fri Jul 14 11:09:01 PDT 2017 /usr/local/ispconfig/server/server.sh: line 22: 10115 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php Fri Jul 14 11:10:01 PDT 2017 Fri Jul 14 11:10:01 PDT 2017 Fri Jul 14 11:10:01 PDT 2017 c1aps1 does not exceed quota qize: 104857600 > 4453884 Fri Jul 14 11:10:01 PDT 2017 c1aps10 does not exceed quota qize: 104857600 > 5035314 Fri Jul 14 11:10:01 PDT 2017 14.07.2017-11:10 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'. Fri Jul 14 11:10:01 PDT 2017 14.07.2017-11:10 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock Fri Jul 14 11:10:01 PDT 2017 finished.
So it seems as if the segfault does not happen on every run. can you see at which minutes it always happens, this can be used to narrow down by looking at the cron plugin schedules to find out which actions get run at that exact time.
Here are the entries from 8:00 AM to current time 12:45 PM today. It seems to be completely random. Code: root@server:/var/log/ispconfig# cat cron.log | grep "Segmentation fault" Fri Jul 14 08:05:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 10324 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php Fri Jul 14 08:08:01 PDT 2017 /usr/local/ispconfig/server/server.sh: line 22: 11063 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php Fri Jul 14 08:24:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 13653 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php Fri Jul 14 08:40:01 PDT 2017 /usr/local/ispconfig/server/server.sh: line 22: 16183 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php Fri Jul 14 08:50:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 17848 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php Fri Jul 14 08:53:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 18614 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php Fri Jul 14 08:59:01 PDT 2017 /usr/local/ispconfig/server/server.sh: line 22: 19528 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php Fri Jul 14 09:09:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 21246 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php Fri Jul 14 09:15:01 PDT 2017 /usr/local/ispconfig/server/server.sh: line 22: 22315 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php Fri Jul 14 09:26:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 24602 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php Fri Jul 14 09:30:01 PDT 2017 /usr/local/ispconfig/server/server.sh: line 22: 24788 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php Fri Jul 14 09:35:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 25611 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php Fri Jul 14 09:36:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 26146 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php Fri Jul 14 10:14:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 32638 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php Fri Jul 14 10:18:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 965 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php Fri Jul 14 10:32:01 PDT 2017 /usr/local/ispconfig/server/server.sh: line 22: 3746 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php Fri Jul 14 10:53:02 PDT 2017 /usr/local/ispconfig/server/server.sh: line 22: 7399 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php Fri Jul 14 11:01:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 9059 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php Fri Jul 14 11:09:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 10114 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php Fri Jul 14 11:09:01 PDT 2017 /usr/local/ispconfig/server/server.sh: line 22: 10115 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php Fri Jul 14 11:19:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 12179 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php Fri Jul 14 11:45:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 18161 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php Fri Jul 14 11:56:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 20515 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php Fri Jul 14 12:24:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 26108 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php Fri Jul 14 12:25:01 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 26159 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php Fri Jul 14 12:28:02 PDT 2017 /usr/local/ispconfig/server/cron.sh: line 17: 26942 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/cron.php Fri Jul 14 12:36:01 PDT 2017 /usr/local/ispconfig/server/server.sh: line 22: 28466 Segmentation fault /usr/bin/php -q -d disable_classes= -d disable_functions= -d open_basedir= /usr/local/ispconfig/server/server.php
Did you look at the syslog file and grep for segfault, to see if other services fail as well from time to time?
Yes, there is nothing. Code: root@server:/var/log# cat syslog.1 | grep "Segmentation fault" root@server:/var/log# cat syslog.1 | grep "Seg fault" root@server:/var/log# cat syslog.1 | grep "segfault" root@server:/var/log# cat syslog.1 | grep "Segmentation" root@server:/var/log# cat syslog.1 | grep "Segfault" root@server:/var/log# cat syslog | grep "Segmentation fault" root@server:/var/log# cat syslog | grep "Seg fault" root@server:/var/log# cat syslog | grep "segfault" root@server:/var/log# cat syslog | grep "Segmentation" root@server:/var/log# cat syslog | grep "Segfault" I also ran each one of the crons manually every minute for ten minutes ( https://www.faqforge.com/linux/debugging-ispconfig-3-server-actions-in-case-of-a-failure/ ) No segmentation fault error displayed during that time.
It might be best to ignore it for now, maybe you find something later, as long as you don't see any faults in the actual system config or services.
