Backup failes with "Allowed memory size exhausted"

Hi there,

on a customers server we run the current ISPC version (3.1.13.p1) on a recent and freshly installed CentOS (7.6.1810).
The server is configured in multi-server mode as a slave and runs the webserver only. The database is running on a separate server.
We faced issues with backups not being listed in the interface below the "Backups" Tab within the website. The Backup mode was set to "Make backups as root user". Anyway, the backup files were successfully created below /var/backups/web...

To analyze the issue, we switched to the other Backup mode "Owned by WEB User as ZIP". First that failed due to missing zip and pigz packages. That was fixed. We then run the cron_debug.php script to get more insight. It nearly seemed to run through (the zip file was built below /tmp) but failed with the following errors:

Code:

sudo php /usr/local/ispconfig/server/cron_debug.php --cronjob=500-backup.inc.php
zip warning: No such file or directory
( ^ x >100 )
.
.
zip warning: No such file or directory
PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 4096 bytes) in /usr/local/ispconfig/server/lib/classes/cron.d/500-backup.inc.php on line 130

Code:

php -v
PHP 7.3.5 (cli) (built: Apr 30 2019 08:37:17) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.5, Copyright (c) 1998-2018 Zend Technologies

The website we tried to backup is about 30GB huge. The server itself has 32GB RAM + modern Xeon based hardware specs.

Code:

tldr;
- Backup in "root" mode works (zips are created and rotated), but will not show up in the ISPC Admin interface for the websites backups
- Backup in "user" mode fails with no trace and a memory issue when debugging with the cron_debug.php

This might be just an issue with the debug tool, but from my understanding, it should not behave any different to the default cron run and the backup should also work with the debug script. I will find out tomorrow after the regular run of the backups tonight.
Anyway, if there are ideas or known issue, I would be glad to know.

Thanks alot, best regards,

Jan

 

Too easy ;-).
Coming from Debian I expected the CLI memory to not be limited at all. But it seems to be a different story with the remi php packages on CentOS...
As usual, thanks Till. Let's see how this works out. I will post back.

 

The increased memory partly did the trick.
Using cron_debug.php it went through this time:

Code:

sudo php /usr/local/ispconfig/server/cron_debug.php --cronjob=500-backup.inc.php
zip warning: No such file or directory
( ^ x 20 )
zip warning: No such file or directory
finished.

But I cannot find any backup files afterwards?!
Backup mode is still "Zip + User"
Both folders
/var/www/[DOMAIN]/backups
as well as
/var/backups/webX/
do not contain any file.

What should be the expected outcome after the backup cron completes? Should anything be in the /var/www/[DOMAIN]/backups at all?

Thanks,
Jan

 

sudo is up and running. I also run the cron_debug using sudo as you can see in the shell output.

 

Hi Till,
finally found some time to dig in again.
Running the cron again as root it went through. But still - with mode "owned by user as zip" - , there is NO file at /var/www/[DOMAIN]/backup/. The file was created at /var/www/backup/web1/. The interface does not list any backups either.

Changing the mode to "backup as root" creates a tar.gz file at the same location. But still nothing within the interface.

Something seems to be very broken here - sadly :-/

Any more ideas?

Best Regards,
Jan

 

Done. No problems:

Code:

 [...] php update.php


--------------------------------------------------------------------------------
 _____ ___________   _____              __ _         ____
|_   _/  ___| ___ \ /  __ \            / _(_)       /__  \
  | | \ `--.| |_/ / | /  \/ ___  _ __ | |_ _  __ _    _/ /
  | |  `--. \  __/  | |    / _ \| '_ \|  _| |/ _` |  |_ |
 _| |_/\__/ / |     | \__/\ (_) | | | | | | | (_| | ___\ \
 \___/\____/\_|      \____/\___/|_| |_|_| |_|\__, | \____/
                                              __/ |
                                             |___/
--------------------------------------------------------------------------------


>> Update

Operating System: Centos 7.6

This application will update ISPConfig 3 on your server.

Shall the script create a ISPConfig backup in /var/backup/ now? (yes,no) [yes]: no

Checking ISPConfig database .. OK
Starting incremental database update.
Loading SQL patch file: /tmp/ispconfig3_install/install/sql/incremental/upd_dev_collection.sql
Reconfigure Permissions in master database? (yes,no) [no]: yes
...
Update finished.

Any other ideas or ways to debug this deeper?

 

@till Any other suggestions how to proceed / debug?

 

Hi @till ,
finally tried out your suggested steps.

Some more context information:
- the dbispconfig_[SLAVE_WEB] and dbispconfig_[SLAVE_DB] databases are filled correctly. The web_backup table contains the correctly created backups (as written, the web page backup still only works in root mode).
- The web_backup table in the dbispconfig (master) DB is empty
- all databases run on the same db server

Both SQL commands work as expected. I used the local mysql connection from the slave using the credentials from the config.inc.php.
The SELECT returned 0 results.
After the INSERT the dummy record is displayed in the Interface.

Code:

MariaDB [dbispconfig]> INSERT INTO web_backup (server_id, parent_domain_id, backup_type, backup_mode, tstamp, filename, filesize) VALUES (2, 1, 'web', 'userzip', 1560784056, 'backup.zip', 99999);
Query OK, 1 row affected (0.00 sec)

MariaDB [dbispconfig]> SELECT * FROM web_backup WHERE server_id = 2;
+-----------+-----------+------------------+-------------+-------------+------------+------------+----------+
| backup_id | server_id | parent_domain_id | backup_type | backup_mode | tstamp     | filename   | filesize |
+-----------+-----------+------------------+-------------+-------------+------------+------------+----------+
|         2 |         2 |                1 | web         | userzip     | 1560784056 | backup.zip | 99999    |
+-----------+-----------+------------------+-------------+-------------+------------+------------+----------+
1 row in set (0.00 sec)

Soo ... any other ideas?

 

*bump*
@till The issue still persist. Should I open up an issue in the tracker for it?