Auto backup not working, but manual yes

Discussion in 'Server Operation' started by Hamid87, Sep 12, 2024.

Tags:
  1. Hamid87

    Hamid87 New Member

    Hi,
    After some days I understood auto backup does not work (manual backup works), so I tried to diagnose and find the problem, after checking the sys_cron table I discovered that none of the corn tasks worked. the last run was 2 months ago.
    I checked the corn job log and it seems to work right:

    Thu 12 Sep 2024 02:32:01 PM +0330 12.09.2024-11:02 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    Thu 12 Sep 2024 02:32:01 PM +0330 12.09.2024-11:02 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    Thu 12 Sep 2024 02:32:01 PM +0330 finished server.php.

    I executed the php cron_debug.php --cronjob=500-backup.inc.php command and the output was "finished cron debug" I checked the sys_cron the date for cronjob_backup updated.

    Do you have any suggestions for solving the issue?
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    Empty (not delete) the sys_cron table in the dbispconfig database. Important: do not mix up with the table named cron, the one you should empty is sys_cron. ISPConfig will rebuild the table then and likely your cronjobs will start working then again.
     
  3. Hamid87

    Hamid87 New Member

    I empty the table sys_cron, and wait 24 hours till cronjobs run, but it is still empty. ISPConfig did not rebuild the table. Is there anything else we can try?
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    Maybe you commented out the ISPConfig cron.sh cronjob (not server.sh) in the root crontab of the server?
     
  5. Hamid87

    Hamid87 New Member

    I checked the /var/log/ispconfig/cron.log it shows cronjob runs every minute.


    Tue 17 Sep 2024 11:10:02 AM +0330 17.09.2024-07:40 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    Tue 17 Sep 2024 11:10:02 AM +0330 17.09.2024-07:40 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    Tue 17 Sep 2024 11:10:02 AM +0330 finished server.php.
    Tue 17 Sep 2024 11:11:01 AM +0330 17.09.2024-07:41 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    Tue 17 Sep 2024 11:11:01 AM +0330 17.09.2024-07:41 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    Tue 17 Sep 2024 11:11:01 AM +0330 finished server.php.
    Tue 17 Sep 2024 11:12:01 AM +0330 17.09.2024-07:42 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    Tue 17 Sep 2024 11:12:01 AM +0330 17.09.2024-07:42 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    Tue 17 Sep 2024 11:12:01 AM +0330 finished server.php.
    Tue 17 Sep 2024 11:13:01 AM +0330 17.09.2024-07:43 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    Tue 17 Sep 2024 11:13:01 AM +0330 17.09.2024-07:43 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    Tue 17 Sep 2024 11:13:01 AM +0330 finished server.php.
    Tue 17 Sep 2024 11:14:01 AM +0330 17.09.2024-07:44 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    Tue 17 Sep 2024 11:14:01 AM +0330 17.09.2024-07:44 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    Tue 17 Sep 2024 11:14:01 AM +0330 finished server.php.
    Tue 17 Sep 2024 11:15:02 AM +0330 17.09.2024-07:45 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    Tue 17 Sep 2024 11:15:02 AM +0330 17.09.2024-07:45 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    Tue 17 Sep 2024 11:15:02 AM +0330 finished server.php.
    Tue 17 Sep 2024 11:16:01 AM +0330 17.09.2024-07:46 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    Tue 17 Sep 2024 11:16:01 AM +0330 17.09.2024-07:46 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    Tue 17 Sep 2024 11:16:01 AM +0330 finished server.php.
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    That's the server.sh cronjob and not the cron.sh. There are two different cronjobs in the root crontab: server.sh is for config changes, and cron.sh is for ISPConfigs internal cronjobs like backups, statistics, log rotation, etc.
     
  7. Hamid87

    Hamid87 New Member

    I checked the crontab, and it seems fine.


    * * * * * /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done
    * * * * * /usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done
    0 0 * * * /usr/local/ispconfig/server/scripts/create_daily_nginx_access_logs.sh &> /dev/null
     
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    ok, and have you run /usr/local/ispconfig/server/cron.sh manually now?
     
  9. Hamid87

    Hamid87 New Member

    Yes, it is work, I checked the corn table it is empty. I feel the problem is the cron table how can I refill it?
     
  10. till

    till Super Moderator Staff Member ISPConfig Developer

    The sys_cron table gets filled when you or the Linux system runs the /usr/local/ispconfig/server/cron.sh script. You can try to run a repair and optimize on all tables of your dbispconfig database e.g. using phpmyadmin, maybe the sys_cron table is broken so that no new data can be saved in it.
     
  11. till

    till Super Moderator Staff Member ISPConfig Developer

    Also, did ou check the right table, as I mentioned above, the table that is relevant for this function is named sys_cron and not cron and you should empty the sys_cron table and not the cron table.
     
  12. Hamid87

    Hamid87 New Member

    Yes, I emptied the sys_cron, but the cron table was empty also.
     
  13. till

    till Super Moderator Staff Member ISPConfig Developer

    The cron table has a different purpose, its for website cronjobs. is the sys_cron table still empty?
     
  14. Hamid87

    Hamid87 New Member

    Yes still empty
     
  15. Hamid87

    Hamid87 New Member

    Solved
     
  16. till

    till Super Moderator Staff Member ISPConfig Developer

    How? It is helpful for others if you explain how you solved it.
     
  17. Hamid87

    Hamid87 New Member

    I feel that maybe repairing or optimizing the table works for me, today I checked the table, and it was filled with data.
     
    till likes this.
  18. HitoDev

    HitoDev Member

    Hello,

    I have thesame problem:

    • All automatic backup stopped
    • ISPC manual backups works
    • All logs are frozen since Sept. 24, no refresh
    • Backup from CLI php /usr/local/ispconfig/server/cron_debug.php --cronjob=500-backup.inc.php works
    • sys_cron table entries has been deleted has been but no changes
    • other CRON than ISPConfig in crontab works

    What can I do ?
     
  19. HitoDev

    HitoDev Member

    UPDATE:
    Finally I think I found the cause of the problem when I tried tu umount /var/backups/ispc (the path target a NAS where remote backup are sent) :
    umount: /var/backups/ispc: target is busy.

    I found unfinished backups, exactly on the date when ISPConfig server logs and scripts stopped working automatically from crontab (https://monovm.com/blog/fix-umount-target-is-busy-error/):

    lsof | grep /var/backups/ispc

    pigz 3943902 root 1w REG 0,46 201624386 78064589 /var/backups/ispc/xxx/xxxx-09-24_01-22.sql.gz
    pigz 3943902 root 5r REG 0,46 10866647826 78064430 /var/backups/ispc/xxx/xxxx-09-24_01-22.sql
    pigz 3943902 3943905 pigz root 1w REG 0,46 201624386 78064589 /var/backups/ispc/xxx/xxxx-09-24_01-22.sql.gz
    pigz 3943902 3943905 pigz root 5r REG 0,46 10866647826 78064430 /var/backups/ispc/xxx/xxxx-09-24_01-22.sql

    Then I suspected these processes where a possible cause for as well for unmount failure and ISConfig server scripts execution failures (from crontab).
    After having killed the processes I was able to unmount /var/backups/ispc and logs restarting to appear in ISPConfig:

    Thu 26 Dec 2024 09:05:02 AM CET finished server.php.
    Thu 26 Dec 2024 09:06:01 AM CET finished server.php.
    Thu 26 Dec 2024 09:07:01 AM CET finished server.php.

    sys_cron table has been updated again.
    I guess tomorrow I'll see last night's backups appear again.
     

Share This Page