Cron errors on Debian 12 multiserver

Discussion in 'Installation/Configuration' started by clixclix, Sep 7, 2023.

Tags:
  1. clixclix

    clixclix Member HowtoForge Supporter

    Hi,
    I just installed ISPConfig 3 in a fresh Debian 12 server, into an existing multiserver setup.
    As soon as I went to see the cron logs (/var/log/ispconfig/cron.log) I found a lot of errors of this kind:
    • ERROR - Replication of datalog_id: 6 failed. Error: (sys_ini) in MySQL server: (localhost) Field 'default_logo' doesn't have a default value # SQL: REPLACE INTO ?? (??,??) VALUES (?,?)
    In order to get over the problem, I tried to remove the STRICT_TRANS_TABLES option from MariaDB configuration.
    This removed that problem, but now I have a lot of other errors on the log, like:
    • ERROR - Replication of datalog_id: 3019 failed. Error: (web_domain) in MySQL server: (localhost) Unknown column 'fastcgi_php_version' in 'field list' # SQL: REPLACE INTO ?? (??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
    • PHP Deprecated: explode(): Passing null to parameter #2 ($string) of type string is deprecated in /usr/local/ispconfig/server/lib/classes/cron.d/100-monitor_hd_quota.inc.php on line 136
    • PHP Deprecated: explode(): Passing null to parameter #2 ($string) of type string is deprecated in /usr/local/ispconfig/server/lib/classes/cron.d/100-monitor_hd_quota.inc.php on line 164
    • PHP Warning: Undefined array key 1 in /usr/local/ispconfig/server/lib/classes/cron.d/100-monitor_mem_usage.inc.php on line 7
    • PHP Deprecated: trim(): Passing null to parameter #1 ($string) of type string is deprecated in /usr/local/ispconfig/server/lib/classes/cron.d/100-monitor_mem_usage.inc.php on line 79
    Also, on the main ISPConfig Admin panel, I have a lot of pending actions.
    What's wrong?
    Thanks a lot for any hint.
     
    clixclix, Sep 7, 2023
    #1
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    Most likely you are using different ISPConfig version in your setup. In a multiserver setup, the ISPConfig version must be identical on all nodes.

    That's not needed and not the source of your issue.
     
    till, Sep 7, 2023
    #2
  3. clixclix

    clixclix Member HowtoForge Supporter

    You're right: when I attached the new server, ISPConfig was not updated to the last version on the master server.
    However, after I updated ISPConfig on the master server, the above cron errors are still generating.
    What do you suggest? Reinstlling everything on the new slave?
    Note: none of the other servers (all of them updated now) are generating errors.
     
    clixclix, Sep 7, 2023
    #3
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    Do you still get "Unknown column" errors, or just the "has no default value" error?
     
    till, Sep 7, 2023
    #4
  5. till

    till Super Moderator Staff Member ISPConfig Developer

    And one more question, do you use MariaDB or MySQL and which version do you use?
     
    till, Sep 7, 2023
    #5
  6. clixclix

    clixclix Member HowtoForge Supporter

    MariaDB version: 10.11.3-MariaDB-1 - Debian 12
    Right now I still have the STRICT_TRANS_TABLES option removed, so I'm not having the "no default value" error.
    The server is still generating, about once every minute, the following errors (here's an excerpt):
    Code:
    gio 7 set 2023, 11:50:01, CEST 07.09.2023-11:50 - ERROR - Replication of datalog_id: 3019 failed. Error: (web_domain) in MySQL server: (localhost) Unknown column 'fastcgi_php_version' in 'field list' # SQL: REPLACE INTO ?? (??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
gio 7 set 2023, 11:50:01, CEST PHP Deprecated:  explode(): Passing null to parameter #2 ($string) of type string is deprecated in /usr/local/ispconfig/server/lib/classes/cron.d/100-monitor_hd_quota.inc.php on line 136
gio 7 set 2023, 11:50:01, CEST PHP Deprecated:  explode(): Passing null to parameter #2 ($string) of type string is deprecated in /usr/local/ispconfig/server/lib/classes/cron.d/100-monitor_hd_quota.inc.php on line 164
gio 7 set 2023, 11:50:01, CEST 07.09.2023-11:50 - ERROR - Error in Replication, changes were not processed.
gio 7 set 2023, 11:50:01, CEST finished server.php.
gio 7 set 2023, 11:50:02, CEST PHP Warning:  Undefined array key 1 in /usr/local/ispconfig/server/lib/classes/cron.d/100-monitor_mem_usage.inc.php on line 79
gio 7 set 2023, 11:50:02, CEST PHP Deprecated:  trim(): Passing null to parameter #1 ($string) of type string is deprecated in /usr/local/ispconfig/server/lib/classes/cron.d/100-monitor_mem_usage.inc.php on line 79
gio 7 set 2023, 11:51:01, CEST 07.09.2023-11:51 - ERROR - Replication of datalog_id: 3019 failed. Error: (web_domain) in MySQL server: (localhost) Unknown column 'fastcgi_php_version' in 'field list' # SQL: REPLACE INTO ?? (??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
gio 7 set 2023, 11:51:01, CEST 07.09.2023-11:51 - ERROR - Error in Replication, changes were not processed.
gio 7 set 2023, 11:51:01, CEST finished server.php.
gio 7 set 2023, 11:52:01, CEST 07.09.2023-11:52 - ERROR - Replication of datalog_id: 3019 failed. Error: (web_domain) in MySQL server: (localhost) Unknown column 'fastcgi_php_version' in 'field list' # SQL: REPLACE INTO ?? (??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
gio 7 set 2023, 11:52:01, CEST 07.09.2023-11:52 - ERROR - Error in Replication, changes were not processed.
gio 7 set 2023, 11:52:01, CEST finished server.php.
gio 7 set 2023, 11:53:01, CEST 07.09.2023-11:53 - ERROR - Replication of datalog_id: 3019 failed. Error: (web_domain) in MySQL server: (localhost) Unknown column 'fastcgi_php_version' in 'field list' # SQL: REPLACE INTO ?? (??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
gio 7 set 2023, 11:53:01, CEST 07.09.2023-11:53 - ERROR - Error in Replication, changes were not processed.
gio 7 set 2023, 11:53:01, CEST finished server.php.
gio 7 set 2023, 11:54:01, CEST 07.09.2023-11:54 - ERROR - Replication of datalog_id: 3019 failed. Error: (web_domain) in MySQL server: (localhost) Unknown column 'fastcgi_php_version' in 'field list' # SQL: REPLACE INTO ?? (??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??,??) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
gio 7 set 2023, 11:54:01, CEST 07.09.2023-11:54 - ERROR - Error in Replication, changes were not processed.
gio 7 set 2023, 11:54:01, CEST finished server.php.
     
    clixclix, Sep 7, 2023
    #6
  7. till

    till Super Moderator Staff Member ISPConfig Developer

    Your master system is ISPCoonfig 3.2.11 as well? Because the column fastcgi_php_version was dropped quite some time ago in a release around February 2021 (see file install/sql/incremental/upd_0092.sql), but your master still seems to have that column in its database and when he tries to copy data to the new slave, this fails as the new system does not have that column.
     
    till, Sep 7, 2023
    #7
  8. michelangelo

    michelangelo Active Member

    Hi Till
    the error lies in the datalog entry itself, because this old datalog record is not compatible anymore with the current table schema.

    edit
    It would be nice to have a solution for this.
    As far as I remember it is not recommended to delete datalog records due to the integrity itself.
     
    michelangelo, Sep 7, 2023
    #8
  9. till

    till Super Moderator Staff Member ISPConfig Developer

    That's what I wanted to say with my post. The reason for that is that the database on the master has a different version than the slave database. Therefore the master tries to create SQL queries for a column that still exists on master but not anymore on the slave.

    The solution is to ensure that the master and slave ISPConfig systems use the same ISPConfig version. The master will not create a new auery with that column when the column has been removed from master database and that happened in February 2021 in install/sql/incremental/upd_0092.sql file.

    So the above error either occurs because the master system uses a different ISPConfig version, or in the past there was an error on that master system which caused the file install/sql/incremental/upd_0092.sql not to be applied on the master.

    Btw. This situation is very common anyway as I try to avoid removing columns in general.
     
    till, Sep 7, 2023
    #9
  10. till

    till Super Moderator Staff Member ISPConfig Developer

    Another possible temporary solution if you use a highly modified system and can't update the master at the moment is that you re-add the missing column on the slave.
     
    till, Sep 7, 2023
    #10
  11. michelangelo

    michelangelo Active Member

    That scenario is not always the case.

    Don't get me wrong, of course it should be checked that the master and slave databases and its table schemas are up-to-date/compatible with each other, but if there is an old datalog record, that contains column data that is not compatible anymore with current column definition, then this will surely end in an error in the replication process as it can be seen for the custom_logo example above, because the value that was saved back then for this column differs to what the current/new column expects.
     
    michelangelo, Sep 7, 2023
    #11
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    Old datalog records will not get replayed when you add a new server unless they have server_id = 0, and we do not remove or repurpose columns in these tables to avoid that problem for this exact reason. The column in question is not in a table that we use server_id = 0 for, its a column in web_domain table. The custom_logo issue is different and not related to table column removals, this seems to be a sql-mode issue indeed, or if we see it the other way around, ispconfig should have a default value setting for this column, which we currently don't have and which should be fixed. I have not seen the custom_logo error on any of my systems yet, so likely it does not occur often, but it needs to be fixed nonetheless, of course.
     
    till, Sep 7, 2023
    #12
  13. clixclix

    clixclix Member HowtoForge Supporter

    Hi everybody, thanks for your useful hints.
    Here's an update.
    In my case, at the time I posted the problem, the Master server had the same ISPConfig version than the slave (3.2.11).
    The fastcgi_php_version column was not present on the master web_domain table.
    In order to get over the problem, as suggested by Till I temporarily added the fastcgi_php_version column on the slave (I also had to add the enable_spdy column).
    After that, the cron job finally was able to flush the job queue.

    I can confirm that the sys_datalog record #3019 on the master server had server_id=0 (and timestamp of 08/04/2013 !)

    Here's an excerpt of that database record:
    Code:
    MariaDB [dbispconfig]> SELECT * FROM `sys_datalog` WHERE `datalog_id`=3019;
| datalog_id | server_id | dbtable    | dbidx         | action | tstamp     | user  | data                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | status | error | session_id |
|       3019 |         0 | web_domain | domain_id:336 | i      | 1365452190 | admin | a:2:{s:3:"new";a:65:{s:9:"domain_id";s:3:"336";s:10:"sys_userid";s:1:"1";s:11:"sys_groupid";s:2:"26";s:13:"sys_perm_user";s:4:"riud";s:14:"sys_perm_group";s:2:"ru";s:14:"sys_perm_other";s:0:"";s:9:"server_id";s:1:"0";s:10:"ip_address";N;s:12:"ipv6_address";N;s:6:"domain";N;s:4:"type";N;s:16:"parent_domain_id";s:1:"0";s:10:"vhost_type";N;s:13:"document_root";s:32:"/var/www/clients/client25/web336";s:10:"web_folder";N;s:11:"system_user";s:6:"web336";s:12:"system_group";s:8:"client25";s:8:"hd_quota";s:1:"0";s:13:"traffic_quota";s:2:"-1";s:3:"cgi";s:1:"y";s:3:"ssi";s:1:"y";s:6:"suexec";s:1:"y";s:9:"errordocs";s:1:"1";s:15:"is_subdomainwww";s:1:"1";s:9:"subdomain";s:4:"none";s:3:"php";s:1:"y";s:4:"ruby";s:1:"n";s:6:"python";s:1:"n";s:4:"perl";s:1:"n";s:13:"redirect_type";s:0:"";s:13:"redirect_path";s:0:"";s:12:"seo_redirect";s:0:"";s:3:"ssl";s:1:"n";s:9:"ssl_state";N;s:12:"ssl_locality";N;s:16:"ssl_organisation";N;s:21:"ssl_organisation_unit";N;s:11:"ssl_country";N;s:10:"ssl_domain";N;s:11:"ssl_request";N;s:8:"ssl_cert";N;s:10:"ssl_bundle";N;s:7:"ssl_key";s:0:"";s:10:"ssl_action";N;s:14:"stats_password";N;s:10:"stats_type";s:9:"webalizer";s:14:"allow_override";s:3:"All";s:17:"apache_directives";N;s:16:"nginx_directives";N;s:18:"php_fpm_use_socket";s:1:"n";s:2:"pm";s:7:"dynamic";s:15:"pm_max_children";s:2:"10";s:16:"pm_start_servers";s:1:"2";s:20:"pm_min_spare_servers";s:1:"1";s:20:"pm_max_spare_servers";s:1:"5";s:23:"pm_process_idle_timeout";s:2:"10";s:15:"pm_max_requests";s:1:"0";s:16:"php_open_basedir";s:217:"/var/www/clients/client25/web336/web:/var/www/clients/client25/web336/tmp:/var/www/forumdac.it/web:/srv/www/forumdac.it/web:/usr/share/php5:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/usr/share/php";s:14:"custom_php_ini";N;s:15:"backup_interval";s:4:"none";s:13:"backup_copies";s:1:"1";s:6:"active";s:1:"y";s:18:"traffic_quota_lock";s:1:"n";s:19:"fastcgi_php_version";N;s:16:"proxy_directives";N;}s:3:"old";a:65:{s:9:"domain_id";N;s:10:"sys_userid";N;s:11:"sys_groupid";N;s:13:"sys_perm_user";N;s:14:"sys_perm_group";N;s:14:"sys_perm_other";s:0:"";s:9:"server_id";N;s:10:"ip_address";N;s:12:"ipv6_address";N;s:6:"domain";N;s:4:"type";N;s:16:"parent_domain_id";N;s:10:"vhost_type";N;s:13:"document_root";N;s:10:"web_folder";N;s:11:"system_user";N;s:12:"system_group";N;s:8:"hd_quota";N;s:13:"traffic_quota";N;s:3:"cgi";N;s:3:"ssi";N;s:6:"suexec";N;s:9:"errordocs";N;s:15:"is_subdomainwww";N;s:9:"subdomain";N;s:3:"php";N;s:4:"ruby";N;s:6:"python";N;s:4:"perl";N;s:13:"redirect_type";s:0:"";s:13:"redirect_path";s:0:"";s:12:"seo_redirect";s:0:"";s:3:"ssl";N;s:9:"ssl_state";N;s:12:"ssl_locality";N;s:16:"ssl_organisation";N;s:21:"ssl_organisation_unit";N;s:11:"ssl_country";N;s:10:"ssl_domain";N;s:11:"ssl_request";N;s:8:"ssl_cert";N;s:10:"ssl_bundle";N;s:7:"ssl_key";s:0:"";s:10:"ssl_action";N;s:14:"stats_password";N;s:10:"stats_type";N;s:14:"allow_override";N;s:17:"apache_directives";N;s:16:"nginx_directives";N;s:18:"php_fpm_use_socket";N;s:2:"pm";N;s:15:"pm_max_children";N;s:16:"pm_start_servers";N;s:20:"pm_min_spare_servers";N;s:20:"pm_max_spare_servers";N;s:23:"pm_process_idle_timeout";N;s:15:"pm_max_requests";N;s:16:"php_open_basedir";N;s:14:"custom_php_ini";N;s:15:"backup_interval";N;s:13:"backup_copies";N;s:6:"active";N;s:18:"traffic_quota_lock";N;s:19:"fastcgi_php_version";N;s:16:"proxy_directives";N;}} | ok     | NULL  |            |
     
    clixclix, Sep 7, 2023
    #13
    till likes this.
  14. till

    till Super Moderator Staff Member ISPConfig Developer

    Ok, thanks for the info. I'll have to dig into the code then, I was not aware that we have server_id 0 records for this table too. But its was long ago :) I guess we have to implement some kind of function to clean up and consolidate server_id 0 records somehow.
     
    till, Sep 7, 2023
    #14
    clixclix likes this.
  15. Paulmw

    Paulmw New Member

    Just ran into this too, adding a new server to an existing cluster. All versions matched.
    So to make the instructions completely clear, add the column with:
    Code:
    ALTER TABLE `web_domain` ADD `fastcgi_php_version` varchar(255) DEFAULT NULL AFTER `traffic_quota_lock`;
    And then after it has synced, remove it with:
    Code:
    ALTER TABLE `web_domain` DROP COLUMN `fastcgi_php_version`;
     
    Paulmw, Mar 25, 2024
    #15

Share This Page