ISPConfig 3 - back-res a Backup and Restore script

Hi Friends!
Running back-res, after db backup, obtain this warnings/errors:

Code:

/sbin/back-res: line 407: [: too arguments
/bin/tar: /www: function "stat" failed: File or directory does not exist
/bin/tar: Exit with failure status based on previous errors

This is the part of the script concerned:

Code:

for i in `echo $DIRECTORIES` ; do
    XX=`echo $i | awk '{gsub("/", "_", $0); print}'`
    YX=`echo $i | awk '{print $1}'`
    fb=`ls $BACKUPDIR | grep ^full$XX-`
        if [ -z $fb ] ; then
            log "No full backup found for $YX. Full backup now!"
            echo > $tmpdir/full-backup$XX.lck
            $TAR $ARG $BACKUPDIR/full$XX-$FDATE.tar.bz2 $YX -X $tmpdir/excluded
            log "Backup of $YX done."
        fi

the 407 line is this:

Code:

if [ -z $fb ] ; then

Why is this happening? Do I have something wrong?

Many many thanks!
Davide

 

If you had read the thread a few back you'd find the same already discussed.
Look here #233

 

Thanks Geekteq, I didn't see the previous post
Now, with the changes, it works fine!
Strange that the developer has not yet corrected the bug! I'll try to report it to him.

 

Hi Friends,
after to have corrected the bug described above,
today (at the first one month day backup time) just a few minutes after starting the "back-res" script the MySQL server hangs:

Code:

service mysql stop

Code:

service mysql start
Job for mysql.service failed. See 'systemctl status mysql.service' and 'journalctl -xn' for details.

Code:

systemctl status mysql.service
● mysql.service - LSB: Start and stop the mysql database server daemon
   Loaded: loaded (/etc/init.d/mysql)
   Active: failed (Result: exit-code) since lun 2018-01-01 15:49:30 CET; 10s ago
  Process: 18210 ExecStop=/etc/init.d/mysql stop (code=exited, status=0/SUCCESS)
  Process: 18261 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE)

gen 01 15:49:30 server /etc/init.d/mysql[19100]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
gen 01 15:49:30 server mysql[18261]: Starting MySQL database server: mysqld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . failed!
gen 01 15:49:30 server systemd[1]: mysql.service: control process exited, code=exited status=1
gen 01 15:49:30 server systemd[1]: Failed to start LSB: Start and stop the mysql database server daemon.
gen 01 15:49:30 server systemd[1]: Unit mysql.service entered failed state.

Then reloading again the MySQL server, everything is broken down correctly.

Has anyone of you encountered similar problems?

Many thanks!

Davide

 

Still a great script after all this time, the only problem I have with it is it does not exclude directories from /var with the latest update, even if specified. log, spool, run, are all added due to the script recursive adding them in the # Add /var excluding subdirectories /var/www and /var/vmail to DIRECTORIES section. (debian 9.4 with version="0.9.6 from 2014-02-04")

Still, great way of keeping daily backups of my sites, I just hate to waste disk space on things I do not need backed up

[edit]

Just fixed this myself, the tar command in debian 9 needs the -X option set before telling it what to backup, or it will include the files it has been told to (expected behaviour for GNU tar, since forever?), so changing the lines (around line 407) from:

Code:

$TAR $ARG $BACKUPDIR/full$XX-$FDATE.tar.bz2 $YX -X $tmpdir/excluded
..
$TAR $ARG $BACKUPDIR/full$XX-$FDATE.tar.bz2 $YX -X $tmpdir/excluded
..
$TAR $NEWER $ARG $BACKUPDIR/$MDATE/i$XX-$FDATE.tar.bz2 $YX -X $tmpdir/excluded 

To:

Code:

$TAR $ARG $BACKUPDIR/full$XX-$FDATE.tar.bz2 -X $tmpdir/excluded $YX
..
$TAR $ARG $BACKUPDIR/full$XX-$FDATE.tar.bz2 -X $tmpdir/excluded $YX
..
$TAR $NEWER $ARG $BACKUPDIR/$MDATE/i$XX-$FDATE.tar.bz2 -X $tmpdir/excluded $YX

fixed the issue.

 

I'm unable to restore /var directory from a date 2018-08-29 Im using back-res version 0.9.6 The program will all show blank when restoring but no files are restored. Also restoring a specific /var/vmail/xxxxx.tld from a particular date does not work. I know there is a backup file .tar.bz2 for that date and for that specific account, i've checked. Full backup restore works. I know that specific date restore works before the update to version 0.9.6

Thanks

 

please have you got anyone right script for backup and restore ispconfig3 ver. 3.0.5.4p5 to last version?

 

Thanks for sharing this as it also fixes the same issue for me.

I did run a backup task and I noticed the following output.

Code:

root@s1:/# /opt/back-res/back-res
[..]
/opt/back-res/back-res: line 290: /tmp/tmpbck/maildata: No such file or directory
[..]
roundcube.session
note     : Table does not support optimize, doing recreate + analyze instead
[..]
mysqldump: Got error: 1044: "Access denied for user 'root'@'localhost' to database 'information_schema'" when using LOCK TABLES
mysqldump: Got error: 1142: "SELECT, LOCK TABLES command denied to user 'root'@'localhost' for table 'accounts'" when using LOCK TABLES
[..]
/bin/tar: /www: Cannot stat: No such file or directory
/bin/tar: Exiting with failure status due to previous errors
[..]
/bin/tar: /var/lib/php7.0-fpm/web2.sock: socket ignored
[..]
/bin/tar: /var/log/mail.info: file changed as we read it

I am a little bit worried about the databases 'information_schema' and 'performance_schema'. The dump does not represent the current database. I am not sure if this would be a real problem as these are system databases and not the databases required/used by the websites and ISPconfig. Would it be an idea to leave these databases out of the backup?

I don't know what to think about the 'tar cannot stat' error. I don't believe the "/www" is required. It did not exist in the root of my system. I just did create it manually and put a place holder file - just to satisfy tar.

Overall it looks like that the script just works fine. At least I have a bunch of usable

And I did notice this file:

Code:

root@s1:/# cat /tmp/tmpbck/maildata
2019-01-09 22:29:40 - back-res - Temp dir /tmp/tmpbck created
[..]
2019-01-09 22:39:02 - back-res - All backup jobs done. Exiting script!

But I did not receive any e-mail. Couldn't find an error in the script itself. I am able to send e-mails manually by reproducing the command from the script myself.

 

You can exclude "/www" from initial script options, useful if you are using /var/www Instead to /www

I think yes, could be fine to exclude 'information_schema' and 'performance_schema', as suggested here: https://stackoverflow.com/questions/14423686/error-1142-select-and-lock-table-commands-denied

No, the best way as suggested above, is exclude it form options script ;-)

I use these parameters for crontab:

Code:

0 0 1 * * /sbin/back-res 2>&1 | mail -s "backup-all-files" my_email@my_domain

 

This script is so helpful - thank you for your contribution to the ISPConfig community! Gratitude!

In my hosting environment some users have very large databases (multiple GB-TB) and should be excluded from the backup. Has anyone extended this script to filter specific databases from the backup? Looking for something akin to what `EXCLUDED` config provides, contextual to MySQL database rather than filesystem.

 

Hi Friends!

I'm try to use back-res on Debian 10, but "tar" seems that is having some problems during backup:

Code:

/usr/bin/tar: The following options were used after any non-optional arguments in archive create or update mode.  These options are positional and affect only arguments that follow them.  Please, rearrange them properly.
/usr/bin/tar: --exclude-from "/tmp/tmpbck/excluded" has no effect
/usr/bin/tar: Failed exit based on previous errors

The problem seems depends from the position where the exclusion command is located:
Ithink so shoud be fine, test please!

Code:

$TAR -X $tmpdir/excluded $ARG $BACKUPDIR/full$XX-$FDATE.tar.bz2 $YX

Thanks

Davide

 

Hi,
1. Does this script backup all files needed to for a disaster recovery? Like web sites, dbs, ispconfig db, mail accounts with their passwords and their mail datas?
2. I want to exlude email data (mails in user account inbox, sent items etc.) so is it ok to exclude vmail in EXCLUDED optipn or shoould I remove vmail from MAILDIR="vmail" ?
3. Why is it backing up many system related folders like: bin, boot etc lib64 etc. ? I mean in case server crashes, I can reinstall OS and Ispconfig and do only user data restore. Is not it better and more stable way for a restore? For this puspose can I exlude all these folders? For a better question, what needs to be backuped for for only user data for ispconfig?
Thanks in advance.

 

I did notice this when running the back-res script...

Code:

/usr/bin/tar: The following options were used after any non-optional arguments in archive create or update mode.  These options are positional and affect only arguments that follow them.  Please, rearrange them properly.
/usr/bin/tar: --exclude-from ‘/tmp/tmpbck/excluded’ has no effect
/usr/bin/tar: Exiting with failure status due to previous errors
/usr/bin/tar: The following options were used after any non-optional arguments in archive create or update mode.  These options are positional and affect only arguments that follow them.  Please, rearrange them properly.
/usr/bin/tar: --exclude-from ‘/tmp/tmpbck/excluded’ has no effect
/usr/bin/tar: Exiting with failure status due to previous errors
/usr/bin/tar: The following options were used after any non-optional arguments in archive create or update mode.  These options are positional and affect only arguments that follow them.  Please, rearrange them properly.
/usr/bin/tar: --exclude-from ‘/tmp/tmpbck/excluded’ has no effect
/usr/bin/tar: Exiting with failure status due to previous errors

Running tar 1.30

Code:

tar --version
tar (GNU tar) 1.30
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.

Any ideas?

 

Hi everyone, I have created a new version of the script which solves several problems.
Can you tell me how to share it here?
I added:
# version 0.10.0 - 2021-04-24 (by Giuseppe Benigno)
# --------------------------
# - Changed mysql import command for import stored functions too
# - Changed variable names for more readability
# - Change compress command line for compatibility with tar new version
# - Change the day of full backup. Now the full backup will be performed whenever there is no one for the current month
# - Change Log file name in $BACKUP_DIR/log/backup-$FULL_DATE.log

Thank you
Giuseppe Benigno

 

Hi everyone,
in the script I posted above, I also prevented it from backing up the whole system, the only directory on the system outside of / var is / etc, due to the configuration files.
I didn't need them but of course you can add as many directories as you want.

My regards
Giuseppe Benigno

 

Hi Th0m, yes, you are right.
I have seen that there are other projects open on github but none have made any changes to the original script.
As soon as I find the time, I create one too.

Thanks
Giuseppe Benigno

 

Great job Giuseppe!
Did you get around to creating a git repo?
I would also add the exclusion of the information_schema and performance_schema databases in the sqldump:

Code:

        for i in $(mysql -u$DB_USER -p$DB_PASSWORD -Bse 'show databases'); do
            if [[ "$i" != "information_schema" && "$i" != "performance_schema" ]]; then
                log "Starting mysqldump $i"
                $(mysqldump -u$DB_USER -p$DB_PASSWORD $i --allow-keywords --comments=false --routines --triggers --add-drop-table > $TMP_DIR/db-$i-$FULL_DATE.sql)
                $TAR $COMPRESS_ARGS $BACKUP_DIR/$MONTH_DATE/db-$i-$FULL_DATE.tar.bz2 -C $TMP_DIR db-$i-$FULL_DATE.sql
                rm -rf $TMP_DIR/db-$i-$FULL_DATE.sql
                log "Dump OK. $i database saved OK!"
            fi
        done