(32)Broken pipe: Error writing to vlogger

Discussion in 'ISPConfig 3 Priority Support' started by francoisPE, May 26, 2022.

Page 1 of 2
  1. francoisPE

    francoisPE Active Member HowtoForge Supporter

    Hello,
    I have a multiserver conf with ispconfig 3.2.8p1.
    On my webserver, I have several php versions: php7.4, 8.0 and 8.1
    I update, upgrade php8.0 and 8.1 package through apt
    Since that, there is one vhost not working : it is using default (php7.4)
    In log I had several different error message :
    Today :
    Code:
    (32)Broken pipe: [client hiddenip:37718]AH00646: Error writing to | /usr/local/ispconfig/server/scripts/vlogger -s access.log -t "%Y%m%d-access.log" /var/log/ispconfig/httpd
    Yesterday :
    Code:
    [Wed May 25 17:35:29.849652 2022] [proxy:error] [pid 246151:tid 140481921013504] (111)Connection refused: AH02454: FCGI: attempt to connect to Unix domain socket /var/lib/php7.4-fpm/web11.sock (*) failed
    I google a bit and see that it could be apache conf.
    Code:
    grep -r web11.sock /etc/apache2/*
/etc/apache2/sites-available/nc.dom.com.vhost:                FastCgiExternalServer /var/www/clients/client1/web11/cgi-bin/php-fcgi-*-80-nc.dom.com -idle-timeout 300 -socket /var/lib/php7.4-fpm/web11.sock -pass-header Authorization  -pass-header Content-Type
/etc/apache2/sites-available/nc.dom.com.vhost:                    #ProxyPassMatch ^/(.*\.php[345]?(/.*)?)$ unix:///var/lib/php7.4-fpm/web11.sock|fcgi://localhost//var/www/clients/client1/web11/web/$1
/etc/apache2/sites-available/nc.dom.com.vhost:                                            SetHandler "proxy:unix:/var/lib/php7.4-fpm/web11.sock|fcgi://localhost"
/etc/apache2/sites-available/nc.dom.com.vhost:                FastCgiExternalServer /var/www/clients/client1/web11/cgi-bin/php-fcgi-*-443-nc.dom.com -idle-timeout 300 -socket /var/lib/php7.4-fpm/web11.sock -pass-header Authorization  -pass-header Content-Type
/etc/apache2/sites-available/nc.dom.com.vhost:                    #ProxyPassMatch ^/(.*\.php[345]?(/.*)?)$ unix:///var/lib/php7.4-fpm/web11.sock|fcgi://localhost//var/www/clients/client1/web11/web/$1
/etc/apache2/sites-available/nc.dom.com.vhost:                                            SetHandler "proxy:unix:/var/lib/php7.4-fpm/web11.sock|fcgi://localhost"
    and
    Code:
    grep -r web11.sock /etc/php
/etc/php/7.4/fpm/pool.d/web11.conf:listen = /var/lib/php7.4-fpm/web11.sock
    I see nothing here that is strange or erroneous.
    Yesterday night I update-alternatives for php even if nothing was wrong.
    I will define a php7.4 specific instead of using default version
    What do you think ?
     
    Last edited: May 26, 2022
    francoisPE, May 26, 2022
    #1
  2. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    Have you tried switching the PHP handler/version for that site and then back?
     
    Th0m, May 26, 2022
    #2
  3. francoisPE

    francoisPE Active Member HowtoForge Supporter

    It could come from that because I change to php8.0 and above grep keep giving php7.4

    sockets location seem unchanged when php version changed
     
    Last edited: May 27, 2022
    francoisPE, May 27, 2022
    #3
  4. Jesse Norell

    Jesse Norell Well-Known Member Staff Member Howtoforge Staff

    Code:
    ls -l /etc/alternatives/*php*
     
    Jesse Norell, May 27, 2022
    #4
  5. francoisPE

    francoisPE Active Member HowtoForge Supporter

    Code:
     ls -l /etc/alternatives/*php*
lrwxrwxrwx 1 root root 15 Aug  3  2021 /etc/alternatives/php -> /usr/bin/php7.4
lrwxrwxrwx 1 root root 31 Aug  3  2021 /etc/alternatives/php.1.gz -> /usr/share/man/man1/php7.4.1.gz
lrwxrwxrwx 1 root root 19 Aug  3  2021 /etc/alternatives/php-cgi -> /usr/bin/php-cgi7.4
lrwxrwxrwx 1 root root 35 Aug  3  2021 /etc/alternatives/php-cgi.1.gz -> /usr/share/man/man1/php-cgi7.4.1.gz
lrwxrwxrwx 1 root root 23 Aug  3  2021 /etc/alternatives/php-cgi-bin -> /usr/lib/cgi-bin/php7.4
lrwxrwxrwx 1 root root 24 May 27 02:09 /etc/alternatives/php-fpm.sock -> /run/php/php8.0-fpm.sock
    bad alternative for socket !

    So i ve just run
    update-alternatives --set php-fpm.sock /run/php/php7.4-fpm.sock
     
    francoisPE, May 27, 2022
    #5
  6. francoisPE

    francoisPE Active Member HowtoForge Supporter

    I am not sure this is this because i have a socket per website with iscp, haven't i ?
     
    francoisPE, May 27, 2022
    #6
  7. till

    till Super Moderator Staff Member ISPConfig Developer

    I don't think that your issue is related to the PHP version that the site runs as vlogger is written in perl and not PHP. Does the issue still occurs? if yes, check if the site has a folder with its domain name in /var/log/ispconfig/httpd/ folder (like all other sites).
     
    till, May 27, 2022
    #7
  8. francoisPE

    francoisPE Active Member HowtoForge Supporter

    There is one and i review it in my site log folder.

    Error.log says
    Code:
    [Fri May 27 17:15:19.620697 2022] [proxy_fcgi:error] [pid 769945:tid 140319450453760] [client 1hidden1.129:60789] AH01067: Failed to read FastCGI header
[Fri May 27 17:15:19.620736 2022] [proxy_fcgi:error] [pid 769945:tid 140319450453760] (104)Connection reset by peer: [client 128.1hidden129:60789] AH01075: Error dispatching request to :                            
[Fri May 27 17:15:19.710419 2022] [proxy:error] [pid 956185:tid 140320222189312] (2)No such file or directory: AH02454: FCGI: attempt to connect to Unix domain socket /var/lib/php7.4-fpm/web11.sock (*) failed
[Fri May 27 17:15:19.710439 2022] [proxy_fcgi:error] [pid 956185:tid 140320222189312] [client 2001:hidden:2:80b3:0:40:52114] AH01079: failed to make connection to backend: httpd-UDS
    I am now with php8 on that website...
    No other options checked (no suexec, no cgi etc)
     
    Last edited: May 27, 2022
    francoisPE, May 27, 2022
    #8
  9. francoisPE

    francoisPE Active Member HowtoForge Supporter

    I found that
    grep web11 /var/log/php*-fpm.log
    a lot of lines like this one !
    Code:
    /var/log/php8.0-fpm.log:[27-May-2022 15:57:39] WARNING: [pool web11] server reached max_children setting (10), consider raising it
    So that I modify parameter (last tab) of my website to 50
    I also add that to php.ini to identify from where issue is coming
    slowlog = /pathtolog/slow.log
    request_slowlog_timeout = 2s
     
    Last edited: May 28, 2022
    francoisPE, May 28, 2022
    #9
  10. francoisPE

    francoisPE Active Member HowtoForge Supporter

    Hello
    I would like to monitor those children process with monit.
    I add those lines in monitrc
    Code:
    check process php8.0-fpm with pidfile /var/run/php/php8.0-fpm.pid      
 group webserver                                      
 start program = "/usr/sbin/service php8.0-fpm start"                  
 stop program = "/usr/sbin/service php8.0-fpm stop"                    
 if cpu > 60% for 1 cycles then alert                                  
 if cpu > 80% for 3 cycles then restart                                
 if totalmem > 25% for 5 cycles then alert                            
 if totalmem > 50% for 5 cycles then restart
 if children > 16 then alert                                
 if children >= 20 then restart                         
 if failed unixsocket /var/run/php/php8.0-fpm.sock then restart        
 if 3 restarts within 5 cycles then timeout
    My questions are
    - in ispc i specify a number of children per website.
    - in monit it seems to be a total number.
    Is there a way in monit to monitor children per website ?
     
    Last edited: May 29, 2022
    francoisPE, May 29, 2022
    #10
  11. Jesse Norell

    Jesse Norell Well-Known Member Staff Member Howtoforge Staff

    It would be simple to monitor the log files for the 'server reached max_children setting' message, which would probably get the same job done for you without having to update your monit config every time you bump the number of php processes for a site.
     
    Jesse Norell, May 31, 2022
    #11
  12. francoisPE

    francoisPE Active Member HowtoForge Supporter

    I finally set a cron job to look at max_children in logs.
    The problem is coming back, and I have no slow.log

    I checked and see that my ispconfig multi server is not working !
    I modified in ispconfig each website > options > Custom php.ini settings to add
    slowlog = /pathtolog/slow.log
    request_slowlog_timeout = 2s
    But, jobs are stuck in the stack ! Red circle not stopping in web interface
    upload_2022-6-2_23-12-2.png

    I move my servers to "debug".
    Here are ispconfig.log

    On web interface server :
    Code:
    02.06.2022-22:53 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
02.06.2022-22:53 - DEBUG [server:177] - Found 2 changes, starting update process.
02.06.2022-22:53 - DEBUG [plugins.inc:118] - Calling function 'server_ip' from plugin 'apache2_plugin' raised by event 'server_update'.
02.06.2022-22:53 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0
02.06.2022-22:53 - DEBUG [apache2 plugin.inc:2469] - Writing the conf file: /etc/apache2/sites-available/ispconfig.conf
02.06.2022-22:53 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
02.06.2022-22:53 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0
02.06.2022-22:53 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
02.06.2022-22:53 - DEBUG [network settings plugin.inc:249] - Network configuration disabled in server settings.
02.06.2022-22:53 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'.
... postfix logs ...
02.06.2022-22:53 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'server_services_plugin' raised by event 'server_update'.
02.06.2022-22:53 - DEBUG [plugins.inc:118] - Calling function 'server_update' from plugin 'webserver_plugin' raised by event 'server_update'.
02.06.2022-22:53 - DEBUG [modules.inc:240] - Processed datalog_id 2360
02.06.2022-22:53 - DEBUG [plugins.inc:118] - Calling function 'server_ip' from plugin 'apache2_plugin' raised by event 'server_update'.
02.06.2022-22:53 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0
02.06.2022-22:53 - DEBUG [apache2 plugin.inc:2469] - Writing the conf file: /etc/apache2/sites-available/ispconfig.conf
02.06.2022-22:53 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
02.06.2022-22:53 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0
02.06.2022-22:53 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
02.06.2022-22:53 - DEBUG [network settings plugin.inc:249] - Network configuration disabled in server settings.
... postfix logs ... 
02.06.2022-22:53 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'server_services_plugin' raised by event 'server_update'.
02.06.2022-22:53 - DEBUG [plugins.inc:118] - Calling function 'server_update' from plugin 'webserver_plugin' raised by event 'server_update'.
02.06.2022-22:53 - DEBUG [modules.inc:240] - Processed datalog_id 2361
02.06.2022-22:53 - DEBUG [services.inc:56] - Calling function 'restartHttpd' from module 'web_module'.
02.06.2022-22:53 - DEBUG [system.inc:2082] - Trying to use Systemd to restart service
02.06.2022-22:53 - DEBUG [system.inc:2399] - safe_exec cmd: systemctl is-enabled 'apache2' 2>&1 - return code: 0
02.06.2022-22:53 - DEBUG [web module.inc:246] - Restarting httpd: systemctl restart apache2.service
02.06.2022-22:53 - DEBUG [services.inc:56] - Calling function 'restartPostfix' from module 'mail_module'.
02.06.2022-22:53 - DEBUG [system.inc:2082] - Trying to use Systemd to restart service
02.06.2022-22:53 - DEBUG [system.inc:2399] - safe_exec cmd: systemctl is-enabled 'postfix' 2>&1 - return code: 0
02.06.2022-22:53 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
02.06.2022-22:54 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
02.06.2022-22:54 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
02.06.2022-22:55 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
02.06.2022-22:55 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
02.06.2022-22:56 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
02.06.2022-22:56 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
02.06.2022-22:57 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
02.06.2022-22:57 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
02.06.2022-22:58 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
02.06.2022-22:58 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
02.06.2022-22:59 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
02.06.2022-22:59 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
02.06.2022-23:00 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
02.06.2022-23:00 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/7.4/fpm/php.ini' - return code: 0
02.06.2022-23:00 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/7.4/cgi/php.ini' - return code: 0
02.06.2022-23:00 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/8.0/cgi/php.ini' - return code: 0
02.06.2022-23:00 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/8.0/fpm/php.ini' - return code: 0
02.06.2022-23:00 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
02.06.2022-23:01 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
02.06.2022-23:01 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
02.06.2022-23:02 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
02.06.2022-23:02 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
02.06.2022-23:03 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
02.06.2022-23:03 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
02.06.2022-23:04 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
02.06.2022-23:04 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
02.06.2022-23:05 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
02.06.2022-23:05 - DEBUG [server:177] - Found 4 changes, starting update process.
02.06.2022-23:05 - DEBUG [plugins.inc:118] - Calling function 'server_ip' from plugin 'apache2_plugin' raised by event 'server_update'.
02.06.2022-23:05 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0
02.06.2022-23:05 - DEBUG [apache2 plugin.inc:2469] - Writing the conf file: /etc/apache2/sites-available/ispconfig.conf
02.06.2022-23:05 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
02.06.2022-23:05 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0
02.06.2022-23:05 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
02.06.2022-23:05 - DEBUG [network settings plugin.inc:249] - Network configuration disabled in server settings.
02.06.2022-23:05 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'.
02.06.2022-23:05 - DEBUG [system.inc:2399] - safe_exec cmd: postconf -e 'smtpd_client_restrictions = check_client_access proxy:mysql:/etc/postfix/mysql-virtual_client.cf
, permit_inet_interfaces, permit_mynetworks, permit_sasl_authenticated, reject_rbl_client zen.spamhaus.org, reject_unauth_pipelining, permit' - return code: 0
02.06.2022-23:05 - DEBUG [system.inc:2399] - safe_exec cmd: postconf -e 'smtpd_helo_restrictions = permit_mynetworks, check_helo_access regexp:/etc/postfix/helo_access,
permit_sasl_authenticated, reject_invalid_helo_hostname, reject_non_fqdn_helo_hostname, check_helo_access regexp:/etc/postfix/blacklist_helo, reject_unknown_helo_hostnam
e, permit' - return code: 0
02.06.2022-23:05 - DEBUG [system.inc:2399] - safe_exec cmd: which 'dovecot' 2> /dev/null - return code: 0
02.06.2022-23:05 - DEBUG [system.inc:2399] - safe_exec cmd: postconf -e 'address_verify_virtual_transport = smtp:[127.0.0.1]:10025' - return code: 0
02.06.2022-23:05 - DEBUG [system.inc:2399] - safe_exec cmd: postconf -e 'address_verify_transport_maps = static:smtp:[127.0.0.1]:10025' - return code: 0
02.06.2022-23:05 - DEBUG [plugins.inc:118] - Calling function 'server_update' from plugin 'rspamd_plugin' raised by event 'server_update'.
02.06.2022-23:05 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'server_services_plugin' raised by event 'server_update'.
02.06.2022-23:05 - DEBUG [plugins.inc:118] - Calling function 'server_update' from plugin 'webserver_plugin' raised by event 'server_update'.
02.06.2022-23:05 - DEBUG [modules.inc:240] - Processed datalog_id 2366
02.06.2022-23:05 - DEBUG [plugins.inc:118] - Calling function 'server_ip' from plugin 'apache2_plugin' raised by event 'server_update'.
02.06.2022-23:05 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0
02.06.2022-23:05 - DEBUG [apache2 plugin.inc:2469] - Writing the conf file: /etc/apache2/sites-available/ispconfig.conf
02.06.2022-23:05 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
02.06.2022-23:05 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0
02.06.2022-23:05 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
02.06.2022-23:05 - DEBUG [network settings plugin.inc:249] - Network configuration disabled in server settings.
02.06.2022-23:05 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'.
02.06.2022-23:05 - DEBUG [system.inc:2399] - safe_exec cmd: postconf -e 'smtpd_client_restrictions = check_client_access proxy:mysql:/etc/postfix/mysql-virtual_client.cf
, permit_inet_interfaces, permit_mynetworks, permit_sasl_authenticated, reject_rbl_client zen.spamhaus.org, reject_unauth_pipelining, permit' - return code: 0
02.06.2022-23:05 - DEBUG [system.inc:2399] - safe_exec cmd: postconf -e 'smtpd_helo_restrictions = permit_mynetworks, check_helo_access regexp:/etc/postfix/helo_access,
permit_sasl_authenticated, reject_invalid_helo_hostname, reject_non_fqdn_helo_hostname, check_helo_access regexp:/etc/postfix/blacklist_helo, reject_unknown_helo_hostnam
e, permit' - return code: 0
02.06.2022-23:05 - DEBUG [system.inc:2399] - safe_exec cmd: which 'dovecot' 2> /dev/null - return code: 0
02.06.2022-23:05 - DEBUG [system.inc:2399] - safe_exec cmd: postconf -e 'address_verify_virtual_transport = smtp:[127.0.0.1]:10025' - return code: 0
02.06.2022-23:05 - DEBUG [system.inc:2399] - safe_exec cmd: postconf -e 'address_verify_transport_maps = static:smtp:[127.0.0.1]:10025' - return code: 0
02.06.2022-23:05 - DEBUG [plugins.inc:118] - Calling function 'server_update' from plugin 'rspamd_plugin' raised by event 'server_update'.
02.06.2022-23:05 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'server_services_plugin' raised by event 'server_update'.
02.06.2022-23:05 - DEBUG [plugins.inc:118] - Calling function 'server_update' from plugin 'webserver_plugin' raised by event 'server_update'.
02.06.2022-23:05 - DEBUG [modules.inc:240] - Processed datalog_id 2367
02.06.2022-23:05 - DEBUG [plugins.inc:118] - Calling function 'server_ip' from plugin 'apache2_plugin' raised by event 'server_update'.
02.06.2022-23:05 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0
02.06.2022-23:05 - DEBUG [apache2 plugin.inc:2469] - Writing the conf file: /etc/apache2/sites-available/ispconfig.conf
02.06.2022-23:05 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
02.06.2022-23:05 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0
02.06.2022-23:05 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
02.06.2022-23:05 - DEBUG [network settings plugin.inc:249] - Network configuration disabled in server settings.
    and my webserver
    Code:
    02.06.2022-22:53 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
02.06.2022-22:53 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
02.06.2022-22:54 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
02.06.2022-22:54 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
02.06.2022-22:55 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
02.06.2022-22:55 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
02.06.2022-22:56 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
02.06.2022-22:56 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
02.06.2022-22:57 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
02.06.2022-22:57 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
02.06.2022-22:58 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
02.06.2022-22:58 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/7.4/fpm/php.ini' - return code: 0
02.06.2022-22:58 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/7.4/cgi/php.ini' - return code: 0
02.06.2022-22:58 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/8.0/cgi/php.ini' - return code: 0
02.06.2022-22:58 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/8.0/fpm/php.ini' - return code: 0
02.06.2022-22:58 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
02.06.2022-22:59 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
02.06.2022-22:59 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
02.06.2022-23:00 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    Sorry for that very long logs, I don't know how to sort !

    Running on both machines /usr/local/ispconfig/server/server.sh
    is saying...
    Code:
    02.06.2022-23:19 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
02.06.2022-23:19 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
finished server.php.
    I don't know where to look at ?
    thank you
     
    francoisPE, Jun 2, 2022
    #12
  13. Jesse Norell

    Jesse Norell Well-Known Member Staff Member Howtoforge Staff

    There are only 2 machines in the install? You can click on the red circle for some indication of what changes are outstanding, as well as under Monitor > Jobqueue, which also shows you for which server the job is outstanding.
     
    Jesse Norell, Jun 3, 2022
    #13
  14. francoisPE

    francoisPE Active Member HowtoForge Supporter

    I have 4 machines.
    Red circle is saying
    upload_2022-6-3_9-22-23.png

    Job queue is showing that everything is blocked on the same server...
    upload_2022-6-3_9-24-19.png

    The settings jobs are probably "turn to Debug mode"
    This is the only thing I did after website update.
     
    francoisPE, Jun 3, 2022
    #14
  15. francoisPE

    francoisPE Active Member HowtoForge Supporter

    At the beginning of this story, because of above issue, I try clean my php conf.
    php8.1 was installed automatically so, few days ago, I uninstalled them.
    As a consequence phpmyadmin and roundcube were uninstalled...
    I reinstalled them and now they are working well.

    Right now, thinking about that, I check if some package could miss.
    I run apt install for all packages linked with php and mariadb.
    and run php -q update.php (without any issue)

    Now,
    I still have the famous circle and same jobs blocked in ispconfig web interface
    But /var/log/ispconfig/ispconfig.log is no more incremented... unchanged from before php -q update.php

    and
    @web1:~# /usr/local/ispconfig/server/server.sh
    gives
    finished server.php.

    @ns1:~# /usr/local/ispconfig/server/server.sh
    gives
    03.06.2022-10:02 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    03.06.2022-10:02 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    finished server.php.
     
    francoisPE, Jun 3, 2022
    #15
  16. francoisPE

    francoisPE Active Member HowtoForge Supporter

    I run php -q update.php on my main server.
    It seems nothing changed.
    Still circle and same number of jobs...

    and
    @ns1:~# /usr/local/ispconfig/server/server.sh
    gives same answer !
     
    francoisPE, Jun 3, 2022
    #16
  17. francoisPE

    francoisPE Active Member HowtoForge Supporter

    Digging, I found that in
    /var/log/mysql/error.log
    Code:
    2022-06-03  9:51:48 565 [Warning] Access denied for user ''@'localhost' (using password: NO)
2022-06-03  9:59:48 1286 [Warning] Access denied for user ''@'localhost' (using password: NO)
2022-06-03 10:07:48 2615 [Warning] Access denied for user ''@'localhost' (using password: NO)
2022-06-03 10:15:48 4593 [Warning] Access denied for user ''@'localhost' (using password: NO)
2022-06-03 10:23:49 5923 [Warning] Access denied for user ''@'localhost' (using password: NO)
2022-06-03 10:31:49 7469 [Warning] Access denied for user ''@'localhost' (using password: NO)
2022-06-03 10:39:49 8818 [Warning] Access denied for user ''@'localhost' (using password: NO)
2022-06-03 10:47:49 9920 [Warning] Access denied for user ''@'localhost' (using password: NO)
2022-06-03 10:55:49 11681 [Warning] Access denied for user ''@'localhost' (using password: NO)
2022-06-03 11:03:49 12910 [Warning] Access denied for user ''@'localhost' (using password: NO)
2022-06-03 11:11:49 13842 [Warning] Access denied for user ''@'localhost' (using password: NO)
2022-06-03 11:19:50 15456 [Warning] Access denied for user ''@'localhost' (using password: NO)
2022-06-03 11:27:50 16731 [Warning] Access denied for user ''@'localhost' (using password: NO)
2022-06-03 11:35:51 18620 [Warning] Access denied for user ''@'localhost' (using password: NO)
2022-06-03 11:43:51 20259 [Warning] Access denied for user ''@'localhost' (using password: NO)
2022-06-03 11:51:51 22223 [Warning] Access denied for user ''@'localhost' (using password: NO)
2022-06-03 11:59:51 23695 [Warning] Access denied for user ''@'localhost' (using password: NO)
2022-06-03 12:07:51 25255 [Warning] Access denied for user ''@'localhost' (using password: NO)
2022-06-03 12:15:51 26805 [Warning] Access denied for user ''@'localhost' (using password: NO)
2022-06-03 12:23:52 27583 [Warning] Access denied for user ''@'localhost' (using password: NO)
2022-06-03 12:31:52 27702 [Warning] Access denied for user ''@'localhost' (using password: NO)
2022-06-03 12:39:52 27815 [Warning] Access denied for user ''@'localhost' (using password: NO)
    On both servers.
    This was before running php -q update.php on my main server
    After message disappears
     
    francoisPE, Jun 3, 2022
    #17
  18. Jesse Norell

    Jesse Norell Well-Known Member Staff Member Howtoforge Staff

    In the master server, check the 'updated' column for all the servers in the 'servers' table and compare it to the 'datalog_id' column of the outstanding jobs in 'sys_datalog' table, maybe those got out of sync and you need to change 'updated' to < 'datalog_id' for them to run.
     
    Jesse Norell, Jun 3, 2022
    #18
  19. francoisPE

    francoisPE Active Member HowtoForge Supporter

    In 'sys_datalog' table, there is 7561 lines !
    If it is outstanding jobs, it is a lot ! If not, I don't know how to filter outstanding...
    In that table, there is something like 150 lines with datalog_id different from 0 !
    All values different from 0 are related to server 1 and all are below updated (don't know if already run ?)

    what should I do ?
     
    francoisPE, Jun 3, 2022
    #19
  20. Jesse Norell

    Jesse Norell Well-Known Member Staff Member Howtoforge Staff

    You can compare the columns to what you see in your job queue, using the server_id, the date, etc. Once you identify the lowest datalog_id that needs run, change the servers.updated column for that server to one less than that.

    And now for the obligatory "backup your data first" warning....
     
    Jesse Norell, Jun 3, 2022
    #20
    Th0m likes this.
Page 1 of 2

Share This Page