ISPConfig 3 - back-res a Backup and Restore script

Discussion in 'Plugins/Modules/Addons' started by go0ogl3, Dec 6, 2009.

  1. danjde

    danjde New Member

    Hi Friends!
    Running back-res, after db backup, obtain this warnings/errors:

    Code:
    /sbin/back-res: line 407: [: too arguments
    /bin/tar: /www: function "stat" failed: File or directory does not exist
    /bin/tar: Exit with failure status based on previous errors

    This is the part of the script concerned:

    Code:
    for i in `echo $DIRECTORIES` ; do
        XX=`echo $i | awk '{gsub("/", "_", $0); print}'`
        YX=`echo $i | awk '{print $1}'`
        fb=`ls $BACKUPDIR | grep ^full$XX-`
            if [ -z $fb ] ; then
                log "No full backup found for $YX. Full backup now!"
                echo > $tmpdir/full-backup$XX.lck
                $TAR $ARG $BACKUPDIR/full$XX-$FDATE.tar.bz2 $YX -X $tmpdir/excluded
                log "Backup of $YX done."
            fi
    

    the 407 line is this:

    Code:
    if [ -z $fb ] ; then
    Why is this happening? Do I have something wrong?

    Many many thanks!
    Davide
     
  2. Geekteq

    Geekteq New Member

    If you had read the thread a few back you'd find the same already discussed.
    Look here #233
     
  3. danjde

    danjde New Member

    Thanks Geekteq, I didn't see the previous post :confused:
    Now, with the changes, it works fine!
    Strange that the developer has not yet corrected the bug! I'll try to report it to him.
     
  4. danjde

    danjde New Member

    Hi Friends,
    after to have corrected the bug described above,
    today (at the first one month day backup time) just a few minutes after starting the "back-res" script the MySQL server hangs:

    Code:
    service mysql stop
    Code:
    service mysql start
    Job for mysql.service failed. See 'systemctl status mysql.service' and 'journalctl -xn' for details.
    Code:
    systemctl status mysql.service
    ● mysql.service - LSB: Start and stop the mysql database server daemon
       Loaded: loaded (/etc/init.d/mysql)
       Active: failed (Result: exit-code) since lun 2018-01-01 15:49:30 CET; 10s ago
      Process: 18210 ExecStop=/etc/init.d/mysql stop (code=exited, status=0/SUCCESS)
      Process: 18261 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE)
    
    gen 01 15:49:30 server /etc/init.d/mysql[19100]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
    gen 01 15:49:30 server mysql[18261]: Starting MySQL database server: mysqld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . failed!
    gen 01 15:49:30 server systemd[1]: mysql.service: control process exited, code=exited status=1
    gen 01 15:49:30 server systemd[1]: Failed to start LSB: Start and stop the mysql database server daemon.
    gen 01 15:49:30 server systemd[1]: Unit mysql.service entered failed state.
    Then reloading again the MySQL server, everything is broken down correctly.

    Has anyone of you encountered similar problems?

    Many thanks!

    Davide
     
  5. yodaofborg

    yodaofborg New Member

    Still a great script after all this time, the only problem I have with it is it does not exclude directories from /var with the latest update, even if specified. log, spool, run, are all added due to the script recursive adding them in the # Add /var excluding subdirectories /var/www and /var/vmail to DIRECTORIES section. (debian 9.4 with version="0.9.6 from 2014-02-04")

    Still, great way of keeping daily backups of my sites, I just hate to waste disk space on things I do not need backed up :)

    [edit]

    Just fixed this myself, the tar command in debian 9 needs the -X option set before telling it what to backup, or it will include the files it has been told to (expected behaviour for GNU tar, since forever?), so changing the lines (around line 407) from:
    Code:
    $TAR $ARG $BACKUPDIR/full$XX-$FDATE.tar.bz2 $YX -X $tmpdir/excluded
    ..
    $TAR $ARG $BACKUPDIR/full$XX-$FDATE.tar.bz2 $YX -X $tmpdir/excluded
    ..
    $TAR $NEWER $ARG $BACKUPDIR/$MDATE/i$XX-$FDATE.tar.bz2 $YX -X $tmpdir/excluded 
    To:
    Code:
    $TAR $ARG $BACKUPDIR/full$XX-$FDATE.tar.bz2 -X $tmpdir/excluded $YX
    ..
    $TAR $ARG $BACKUPDIR/full$XX-$FDATE.tar.bz2 -X $tmpdir/excluded $YX
    ..
    $TAR $NEWER $ARG $BACKUPDIR/$MDATE/i$XX-$FDATE.tar.bz2 -X $tmpdir/excluded $YX
    fixed the issue.
     
    Last edited: May 18, 2018
    30uke, Gruppo Pegaso and ahrasis like this.
  6. toyinal

    toyinal Member

    I'm unable to restore /var directory from a date 2018-08-29 Im using back-res version 0.9.6 The program will all show blank when restoring but no files are restored. Also restoring a specific /var/vmail/xxxxx.tld from a particular date does not work. I know there is a backup file .tar.bz2 for that date and for that specific account, i've checked. Full backup restore works. I know that specific date restore works before the update to version 0.9.6

    Thanks
     
  7. dejvic

    dejvic Member

    please have you got anyone right script for backup and restore ispconfig3 ver. 3.0.5.4p5 to last version?
     
  8. 30uke

    30uke Active Member HowtoForge Supporter

    Thanks for sharing this as it also fixes the same issue for me.

    I did run a backup task and I noticed the following output.

    Code:
    root@s1:/# /opt/back-res/back-res
    [..]
    /opt/back-res/back-res: line 290: /tmp/tmpbck/maildata: No such file or directory
    [..]
    roundcube.session
    note     : Table does not support optimize, doing recreate + analyze instead
    [..]
    mysqldump: Got error: 1044: "Access denied for user 'root'@'localhost' to database 'information_schema'" when using LOCK TABLES
    mysqldump: Got error: 1142: "SELECT, LOCK TABLES command denied to user 'root'@'localhost' for table 'accounts'" when using LOCK TABLES
    [..]
    /bin/tar: /www: Cannot stat: No such file or directory
    /bin/tar: Exiting with failure status due to previous errors
    [..]
    /bin/tar: /var/lib/php7.0-fpm/web2.sock: socket ignored
    [..]
    /bin/tar: /var/log/mail.info: file changed as we read it
    
    I am a little bit worried about the databases 'information_schema' and 'performance_schema'. The dump does not represent the current database. I am not sure if this would be a real problem as these are system databases and not the databases required/used by the websites and ISPconfig. Would it be an idea to leave these databases out of the backup?

    I don't know what to think about the 'tar cannot stat' error. I don't believe the "/www" is required. It did not exist in the root of my system. I just did create it manually and put a place holder file - just to satisfy tar.

    Overall it looks like that the script just works fine. At least I have a bunch of usable

    And I did notice this file:
    Code:
    root@s1:/# cat /tmp/tmpbck/maildata
    2019-01-09 22:29:40 - back-res - Temp dir /tmp/tmpbck created
    [..]
    2019-01-09 22:39:02 - back-res - All backup jobs done. Exiting script!
    
    But I did not receive any e-mail. Couldn't find an error in the script itself. I am able to send e-mails manually by reproducing the command from the script myself.
     
  9. danjde

    danjde New Member

    You can exclude "/www" from initial script options, useful if you are using /var/www Instead to /www

    I think yes, could be fine to exclude 'information_schema' and 'performance_schema', as suggested here: https://stackoverflow.com/questions/14423686/error-1142-select-and-lock-table-commands-denied

    No, the best way as suggested above, is exclude it form options script ;-)

    I use these parameters for crontab:

    Code:
    0 0 1 * * /sbin/back-res 2>&1 | mail -s "backup-all-files" my_email@my_domain
    
     
    ahrasis likes this.
  10. This script is so helpful - thank you for your contribution to the ISPConfig community! Gratitude!

    In my hosting environment some users have very large databases (multiple GB-TB) and should be excluded from the backup. Has anyone extended this script to filter specific databases from the backup? Looking for something akin to what `EXCLUDED` config provides, contextual to MySQL database rather than filesystem.
     
  11. danjde

    danjde New Member

    Hi Friends!

    I'm try to use back-res on Debian 10, but "tar" seems that is having some problems during backup:

    Code:
    /usr/bin/tar: The following options were used after any non-optional arguments in archive create or update mode.  These options are positional and affect only arguments that follow them.  Please, rearrange them properly.
    /usr/bin/tar: --exclude-from "/tmp/tmpbck/excluded" has no effect
    /usr/bin/tar: Failed exit based on previous errors
    The problem seems depends from the position where the exclusion command is located:
    Ithink so shoud be fine, test please!

    Code:
    $TAR -X $tmpdir/excluded $ARG $BACKUPDIR/full$XX-$FDATE.tar.bz2 $YX
    Thanks

    Davide
     
  12. ozgurerdogan

    ozgurerdogan Member

    Hi,
    1. Does this script backup all files needed to for a disaster recovery? Like web sites, dbs, ispconfig db, mail accounts with their passwords and their mail datas?
    2. I want to exlude email data (mails in user account inbox, sent items etc.) so is it ok to exclude vmail in EXCLUDED optipn or shoould I remove vmail from MAILDIR="vmail" ?
    3. Why is it backing up many system related folders like: bin, boot etc lib64 etc. ? I mean in case server crashes, I can reinstall OS and Ispconfig and do only user data restore. Is not it better and more stable way for a restore? For this puspose can I exlude all these folders? For a better question, what needs to be backuped for for only user data for ispconfig?
    Thanks in advance.
     
    Last edited: Sep 2, 2020
  13. 30uke

    30uke Active Member HowtoForge Supporter

    I did notice this when running the back-res script...
    Code:
    /usr/bin/tar: The following options were used after any non-optional arguments in archive create or update mode.  These options are positional and affect only arguments that follow them.  Please, rearrange them properly.
    /usr/bin/tar: --exclude-from ‘/tmp/tmpbck/excluded’ has no effect
    /usr/bin/tar: Exiting with failure status due to previous errors
    /usr/bin/tar: The following options were used after any non-optional arguments in archive create or update mode.  These options are positional and affect only arguments that follow them.  Please, rearrange them properly.
    /usr/bin/tar: --exclude-from ‘/tmp/tmpbck/excluded’ has no effect
    /usr/bin/tar: Exiting with failure status due to previous errors
    /usr/bin/tar: The following options were used after any non-optional arguments in archive create or update mode.  These options are positional and affect only arguments that follow them.  Please, rearrange them properly.
    /usr/bin/tar: --exclude-from ‘/tmp/tmpbck/excluded’ has no effect
    /usr/bin/tar: Exiting with failure status due to previous errors
    
    Running tar 1.30
    Code:
    tar --version
    tar (GNU tar) 1.30
    Copyright (C) 2017 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    
    Written by John Gilmore and Jay Fenlason.
    Any ideas?
     
  14. Giuseppe Benigno

    Giuseppe Benigno New Member

    Hi everyone, I have created a new version of the script which solves several problems.
    Can you tell me how to share it here?
    I added:
    # version 0.10.0 - 2021-04-24 (by Giuseppe Benigno)
    # --------------------------
    # - Changed mysql import command for import stored functions too
    # - Changed variable names for more readability
    # - Change compress command line for compatibility with tar new version
    # - Change the day of full backup. Now the full backup will be performed whenever there is no one for the current month
    # - Change Log file name in $BACKUP_DIR/log/backup-$FULL_DATE.log

    Thank you
    Giuseppe Benigno
     

    Attached Files:

    Last edited: Apr 26, 2021
    Farsus, ahrasis, till and 1 other person like this.
  15. Giuseppe Benigno

    Giuseppe Benigno New Member

    Hi everyone,
    in the script I posted above, I also prevented it from backing up the whole system, the only directory on the system outside of / var is / etc, due to the configuration files.
    I didn't need them but of course you can add as many directories as you want.

    My regards
    Giuseppe Benigno
     
  16. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    Maybe a GitHub repo can be created for the script?
     
    Giuseppe Benigno and ahrasis like this.
  17. Giuseppe Benigno

    Giuseppe Benigno New Member

    Hi Th0m, yes, you are right.
    I have seen that there are other projects open on github but none have made any changes to the original script.
    As soon as I find the time, I create one too.

    Thanks
    Giuseppe Benigno
     
    ahrasis likes this.
  18. miguelal

    miguelal New Member

    Great job Giuseppe!
    Did you get around to creating a git repo?
    I would also add the exclusion of the information_schema and performance_schema databases in the sqldump:
    Code:
            for i in $(mysql -u$DB_USER -p$DB_PASSWORD -Bse 'show databases'); do
                if [[ "$i" != "information_schema" && "$i" != "performance_schema" ]]; then
                    log "Starting mysqldump $i"
                    $(mysqldump -u$DB_USER -p$DB_PASSWORD $i --allow-keywords --comments=false --routines --triggers --add-drop-table > $TMP_DIR/db-$i-$FULL_DATE.sql)
                    $TAR $COMPRESS_ARGS $BACKUP_DIR/$MONTH_DATE/db-$i-$FULL_DATE.tar.bz2 -C $TMP_DIR db-$i-$FULL_DATE.sql
                    rm -rf $TMP_DIR/db-$i-$FULL_DATE.sql
                    log "Dump OK. $i database saved OK!"
                fi
            done
     

Share This Page