Sites - automatic backup

We have been using ISPConfig for quite a while without any issues. Since the latest update of ISPConfig (perhaps conincidence, not reason?) the automatic backup function does not work anymore. The main system is Ubuntu 24.04.2 LTS (Noble Numbat)) ISPConfig 3.3.0p2 with latest updates always (unattended). We have been using the backup function in ISPConfig (sites -> <site> -> tab backup) for years without an issue. After updating ispconfig to 3.3.0p2 we noticed that no new backups were written. We checked the disk space and the directory /var/backup/ for permissions

Code:

root@server:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           6.3G  1.8M  6.3G   1% /run
/dev/sda2       2.7T  2.1T  488G  82% /
tmpfs            32G  4.6M   32G   1% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           6.3G   16K  6.3G   1% /run/user/0

root@server:~# ls -la /var/backup/
total 152
drwxr-x--- 38 root root 4096 Jul 11 10:29 .
drwxr-xr-x 16 root root 4096 May 25 12:28 ..
drwxr-x---  2 root root 4096 Feb 26  2024 dumps
drwx------  2 root root 4096 Aug  4 05:00 ispconfig
…
drwx------  2 root root 4096 May 25 13:17 ispconfig_server.manager56.com_2025-05-25_13-17
drwx------  2 root root 4096 Jun 24 15:44 ispconfig_server.manager56.com_2025-06-24_13-44
…
drwxr-x---  2 root root 4096 Jul 30 23:59 web8
…

After deleting old backups via web interface, a manual backup was successfully created. We then configured "no backup", saved and reconfigured "daily backup, retain 8 copies".
Unfortunately, no backups are created automatically anymore, manual backups can be triggered. Can anybody point out what we are doing wrong?

 

Thanks for the fast reply! Here is the output:

Code:

root@server:~# crontab -l
43 9 16 * * /etc/init.d/le_ispc_pem.sh
* * * * * /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done
* * * * * /usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done

The "cron.sh" job is present. The directory /usr/local/ispconfig/server/ looks alright to me:

Code:

root@server:~# ls -la /usr/local/ispconfig/server/
total 100
drwxr-x--- 14 root root 4096 Aug  2 05:00 .
drwxr-xr-x  6 root root 4096 May 13 11:01 ..
drwxrwx---  2 root root 4096 Apr 30  2020 aps_packages
-rwxr-x---  1 root root 3578 Jun 24 15:45 backup-now.php
drwxr-x---  3 root root 4096 Jul 25  2024 cli
drwxr-x---  5 root root 4096 May 13 11:01 conf
drwxr-x---  6 root root 4096 Apr 30  2020 conf-custom
-rwxr-x---  1 root root 5399 Jun 24 15:45 cron.php
-rwxr-x---  1 root root  563 Jun 24 15:45 cron.sh
-rwxr-x---  1 root root   81 Jun 24 15:45 cron_daily.php
-rwxr-x---  1 root root   74 Jun 24 15:45 cron_daily.sh
-rwxr-x---  1 root root 2841 Jun 24 15:45 cron_debug.php
drwxr-x---  3 root root 4096 Jul 21  2022 lib
drwxr-x---  2 root root 4096 May 13 11:01 mods-available
drwxr-x---  2 root root 4096 Apr 30  2020 mods-core
drwxr-x---  2 root root 4096 May 13 11:01 mods-enabled
drwxr-x---  2 root root 4096 May 13 11:01 plugins-available
drwxr-x---  2 root root 4096 May 13 11:01 plugins-enabled
drwxr-x---  2 root root 4096 Jun 24 15:45 scripts
-rwxr-x---  1 root root 9441 Jun 24 15:45 server.php
-rwxr-x---  1 root root  755 Jun 24 15:45 server.sh
drwxr-x---  2 root root 4096 Aug  5 10:31 temp

 

Hi Till, I am not familiar with phpmyadmin, sorry for that. According to your hint I would issue the following commands:

Code:

root@server:~# mysql -uroot -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 1025019
Server version: 10.11.13-MariaDB-0ubuntu0.24.04.1 Ubuntu 24.04

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> use dbispconfig;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [dbispconfig]> TRUNCATE TABLE sys_cron;
MariaDB [dbispconfig]> exit
Bye

(This was just a dry run, not yet executed…)

 

Before truncating I deleted the backup jobs within the respective sites. Thereafter I executed the following:

Code:

root@server:~# mysql -uroot -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 1027376
Server version: 10.11.13-MariaDB-0ubuntu0.24.04.1 Ubuntu 24.04
…
MariaDB [(none)]> use dbispconfig;
…
Database changed
MariaDB [dbispconfig]> TRUNCATE TABLE sys_cron;
Query OK, 0 rows affected (0.182 sec)

MariaDB [dbispconfig]> ^DBye
root@server:~#

I will reconfigure the backup jobs now and keep you posted about the result. Thanks a lot for your fast support!

 

Unfortunately truncating the DB table did not change the situation, no automatic backups are written. Other ideas where to check?
What about a force update install with reconfiguring services…?

 

I did a force update and opposite to the default let it reconfigure permissions in master the database. Other than that, have you got another idea?

 

I started the job manually

Code:

root@server:/usr/local/ispconfig/server# php cron_debug.php --cronjob=500-backup.inc.php

There is an exitcode 1, but the script continues…

Code:

07.08.2025-11:01 - DEBUG [system.inc:2539] - safe_exec cmd: which 'pigz' 2> /dev/null - return code: 1

I will wait for the script to finish.

 

Don't know if it has anything to do with the issue, but it seems like "pigz" is not installed on your system. It is not installed on my fresh ISPConfig Dev VM either. I think it won't hurt to install it and see if anything changes, simply run "apt install pigz".

 

I already did after seeing the error, but the script is still running. Looks like it stalls at a certain point. Let's wait…

 

I became impatient and interrupted the cron_debug.php job which was started at 07.08.2025-11:01 after 10 hours. Then I deleted all old backups and the partial backup from the directories in /var/backup/web[ID]/*. I also repeated the truncate of the sys_cron table and configured new backup jobs via web UI. Since I had been already installing the parallel gzip package pigz after noticing the exitcode while running cron_debug.php I see that now everything is working fine, automatic backups are created again.

So the solution for our problem most likely was installing pigz, probably clearing the sys_cron table helped.

Thanks a lot to till and pyte for paving the way

 

Addendum with real cause of missing backups:
I have further investigated to find the real reason for the missing automatic backups because after 2 days we were missing backups again. We found one website where the user configured their backup themselves and they had a directory size of 60 GB that went into the archive. I excluded some of the user's own backup subdirectories to boil the whole thing down to about 40 GB of total backup size. The backup took still 7:25 h while its MySQL schema backup lasted 14 min. only.

All other sites have reasonable sizes and the total backup time for them is a mere 30 minutes.
So in the end the huge size of a website with a backup that lasted way too long prevented the other sites from being written automatically within a 24 h period. Anyway the debugging brought the missing pigz to light and the case expanded our knowledge of ISPConfig.

Thanks again