Few errors while executing the back-res script Hello, I'm trying to use the back-res script. I'm getting a few errors and it doesn't work for me. I'm using ISPConfig 3 on Ubuntu. First I got this: Code: mysqldump: Got error: 1044: Access denied for user 'root'@'localhost' to database 'information_schema' when using LOCK TABLES I checked the rights and the root user has all rights globally, also for LOCK TABLES. I added "--single-transaction" to the mysqldump link in the script. But I find the error message strange because all the necessary rights are provided. I also added LOCK TABLES specifically for the information_schema database but I got the same message. After adding the single transaction line the script would continue but then I got this message. Code: /bin/tar: /root.: Cannot stat: No such file or directory /bin/tar: Exiting with failure status due to previous errors Any thoughts? It's a pretty basic ISPConfig setup. Only a few domains for private use. I used the HowtoForge guido for installing ISPConfig 3 on Ubuntu.
update I was able to get the backup-res to restore the directories. I had to mkdir /tmp/tmpbck I am running the script as root. Don't know why it will not create the /tmp directories. Now when I try to run as root ./backup-res db all 2011-04-01 this is the error I get date: invalid date `11/' ./backup-res: line 448: [: : integer expression expected here is line 448 if [ "$RD" -gt "$ID" ] ; then Restoring all mysql databases from date 2011-04-01 to local server: mysql: unknown option '-x' mysql: unknown option '-x' mysql: unknown option '-x' mysql: unknown option '-x' mysql: unknown option '-x' mysql: unknown option '-x' mysql: unknown option '-x' mysql: unknown option '-x' mysql: unknown option '-x' mysql: unknown option '-x' All restore jobs done! Database all restored to date 2011-04-01! ./backup-res: line 573: /tmp/tmpbck/maildata: No such file or directory The data bases are not restored. Thanks so much for you help and patient with me and the others.
Some errors Hello, - for the first error please check your mysql version, use latest available for Ubuntu. Maybe this can help: http://bugs.mysql.com/bug.php?id=21527 For what I know there is no need to modify the database dumps lines in the script... I will make some checks later this weekend if I'll have some spare time. - for the second error please check your dirs you want to backup, you have some dot "." in the message and the script complains about not finding the "/root." directory, not the "/root"... If you have more problems, or you can figure out a way to get rid of the mysql error please return here so your message helps others too.
Date problems Hello, I think there is a date formatting problem on your system... Can you run the next commands and post the output here: Code: date +%d date +%F date +%Y-%m The output should be for example: Code: user@main:~$ date +%d 15 user@main:~$ date +%F 2011-04-15 user@main:~$ date +%Y-%m 2011-04 gabi@main:~$ The last mysql errors are because of the date problems. With the tempdir, I don't know why you get the error, there is the "check_tempdir" function which should create the tempdir if it's not there already and it's only failling if the /tmp dir does not exist (it's not a dir). I can put the "-p" option to mkdir but I don't know a linux computer whitout /tmp... so it's like a security option... I'm open to suggestions
Posting the date code Here is the results root@c3po:~# date +%d 15 root@c3po:~# date +%F 2011-04-15 root@c3po:~# date +%Y-%m 2011-04 Here is the result of the ls -l of /tmp drwxrwxrwx 6 root root 4096 2011-04-15 14:35 tmp
Hello, The date results seems ok, but your temp dir does not have the sticky bit set (permissions like drwxrwxrwxt). I'm not sure this can influence the script so much... On a second review of your errors... - do you have any dir with "-" in the name, excepting the date? This can be a problem because the date of the backup archives is extracted from their names. (Please read the detailed description) - for the mysql error - do you have a line in your script identical with the one below? Code: mysql -u$dbuser -p$dbpassword $rdb < $TAR -xvjp $BACKUPDIR/$YDATE-$MDATE/$i If you do then the error is because $TAR variable is empty, which means the script can't find tar on your system (which tar) If you don't have the line identical with the one above please get the latest version of the script.
More info on error I changed the sticky bit this morning drwxrwxrwt 5 root root 4096 2011-04-17 08:01 still get the error touch: cannot touch `/tmp/tmpbck/datestart': No such file or directory touch: cannot touch `/tmp/tmpbck/dateend': No such file or directory find: `/tmp/tmpbck/datestart': No such file or directory find: `/tmp/tmpbck/datestart': No such file or directory find: `/tmp/tmpbck/dateend': No such file or directory When I tried to do the ./backup-res db all 2011-04-01 I get this date: invalid date `11/' ./backup-res: line 448: [: : integer expression expected this is line 448 if [ "$RD" -gt "$ID" ] ; then I am not sure what the `11 is all about I will post on some of the other sites about the making and deleting of the /tmp/tmpbck and see what I can find Here is the line in my backup-res mysql -u$dbuser -p$dbpassword $rdb < $TAR -xvjp $BACKUPDIR/$YDATE-$MDATE/$i here is the only part of the script I made changes to and I do have the latest ############################### ### Begin variables section ### ############################### # Change the variables below to fit your computer/backup COMPUTER=`cat /etc/hostname | awk 'NR==1{print $1}'` # name of this computer DIRECTORIES="/var/lib/mysql /var/www /var/vmail /etc" # directories to backup EXCLUDED="/bck /tmp /dev /proc /sys /srv /media /var/adm /var/cache /var/run /var/lock /lib/init/rw /var/tmp /var/log/verlihub /var/lib/amavis /var/amavis /var/spool/postfix/p* /var/spool/postfix/var *.pid *.lock *.lck" # exclude those dir's and files BACKUPDIR="/bck/$COMPUTER" # where to store the backups dbuser="root" # database user dbpassword="xxxxxxxxxxxx" # database password email="[email protected]" # mail for the responsible person TAR=`which tar` # name and location of tar ARG="-cjpSPf" #sparse # tar arguments P = removed /. EARG="-xjpf" # tar extract arguments P = removed / tmpdir="/tmp/tmpbck" # temp dir for database dump and other stuff del_en="yes" # Enable delete of files if used space percent > than $maxp (yes or anything else) maxp="85" # Max percent of used space before start of delete hm="2359" # last minute of the day = last minute of the restored backup of the day restored ################################### ### End user editable variables ### ################################### as for the "-" in the file names, in /var/lib/mysql there are not dir or files with that in it they all have "_". In the /var/www folder there are folders with that "-" in it, but those files restored. in the /var/www there are php-cgi-scripts and php-fcgi-scripts that I gues were made by ISPconfig 3 install... Thanks again for your help...is there anything else you can think of I can get that may help to solve the issue.
Dirty fixes Hello, a quick fix for the tmpdir is to change the line: Code: mkdir $tmpdir in Code: mkdir -p $tmpdir If the database names contains the "-" or the "_" characters is another thing wich can cause problems. If the date from the filename for the databases is not in form 2011-04-17 there is another point of failure. But still the error from the 448 line is because $RD it's not an integer. RD should be the first backup date calculated in epoch, Code: date --date "$resdate" +%s So if all of the date parts are ok (previous commands you run), why RD is not? Maybe $redate is not what it should be... And $resdate is obtained from those lines: Code: year=`ls -ctF $BACKUPDIR | grep -v ^log/ | tail -n 1 | cut -d "-" -f 2` md=`ls -ctF $BACKUPDIR | grep -v ^log/ | tail -n 1 | cut -d "-" -f 3` day=`ls -ctF $BACKUPDIR | grep -v ^log/ | tail -n 1 | cut -d "-" -f 4 | cut -d "." -f 1` resdate=$year$md$day So the problem it's because something strage happens when we extract the date parts from existing files. I will list here the "debug procedure", hope we'll be able to solve this issue now: - We can start with the $BACKUPDIR which is composed from the /etc/hostname file. - If that is correct we can move to the next step, Code: ls -ctF $BACKUPDIR command which you can run in the comand line, like Code: ls -ctF /bck/your_hostname where your_hostname is the one from the /etc/hostname - If that list seems like: Code: 2011-04/ full_var-2011-04-15.tar.bz2 full_sbin-2011-04-15.tar.bz2 full_lib64-2011-04-15.tar.bz2 full_home-2011-04-15.tar.bz2 full_boot-2011-04-15.tar.bz2 log/ full_www-2011-04-15.tar.bz2 full_usr-2011-04-15.tar.bz2 full_root.-2011-04-15.tar.bz2 full_lib-2011-04-15.tar.bz2 full_etc-2011-04-15.tar.bz2 full_bin-2011-04-15.tar.bz2 we can move to the next step: - the first pipe Code: ls -ctF /bck/your_hostname | grep -v ^log/ the result should be like Code: 2011-04/ full_www-2011-04-15.tar.bz2 full_var-2011-04-15.tar.bz2 full_usr-2011-04-15.tar.bz2 full_sbin-2011-04-15.tar.bz2 full_root.-2011-04-15.tar.bz2 full_lib64-2011-04-15.tar.bz2 full_lib-2011-04-15.tar.bz2 full_home-2011-04-15.tar.bz2 full_etc-2011-04-15.tar.bz2 full_boot-2011-04-15.tar.bz2 full_bin-2011-04-15.tar.bz2 - then we add the next pipe Code: ls -ctF /bck/your_hostname | grep -v ^log/ | tail -n 1 and the result should be the first file, like: Code: full_bin-2011-04-15.tar.bz2 - we then proceed to the last pipe, Code: ls -ctF /bck/your_hostname | grep -v ^log/ | tail -n 1 | cut -d "-" -f 2 which should give us the year like Code: 2011 Hope you can follow those and post where the output is different so we can see where the problem is.
Awesome script, really. I read a post here about some user asking for "remote backup". Ok, it should be possible using sshfs... ______________________________________________________ In debian (using "test" user and remote.domain.com as fqdn of the remote computer as example) apt-get -y install sshfs sudo -i mkdir /localmountdir adduser test fuse sshfs [email protected]:/remotefoldertomount /localmountdir -p 22 next, you must specify "/localmountdir/" as backup dir in go0ogl3's script. _____________________________________________________ I can access the mounted fs with this, but I can't write on it (permission denied)... Maybe it is not working because I don't allow access in ssh for "root user". I have to make a few tests using root as user to login in ssh... if someone of you could help me with this, maybe we can use this script to make a backup on a remote computer. Thanks to all, edit: ok, as root it works perfect (you must add root to fuse group, don't use sudo, login as root in ssh and sshfs, it works for perfect). I need to make it work using another user, because I don't want to allow root login in ssh/sshfs, please help! ) edit2: I made it work without root access. You must chown the remote folder firts (in the server) and next you must mount it on the "local" server. There is no way to chown files on mounter remote folder, but it is not a problem for this scripts, because you don't need any special owner for the tar.gz files... remember to "restore" the files in a folder OUT of the mounter fs, in order to keep original ownership of the files. Regards
Backp-res can be use for ISPConfig 2 too? I read the thread and seems that everybody accept that it is a very nice script and working well. Can be that script used on ISPConfig 2? I'm looking to have a production server ( ubuntu 10.10 ) and a test server, same configuration with same ISPconfig version and every nigt the script to make a full backup( incremental ) , send it to test server and other script to restore the backup . If anything hapent to the production server I have only to change the ip adreses . Anybody have that scenario working?
I think that it should work, but maybe rsync is better for what you want (but you will not have an incremental backup)
Hi, great script. I would suggest using the --hex-blob option and the field and database name quoting when using mysqldump.
ISPC 2 backup and restore Thank you for suggestion, do you have any HOWTO steps for that. I'm sure that it is many users will like to have an automatic backup and restore module. Thanks,
I just meant the back-res script uses mysqldump without the --hex-blob and --quote-names (-Q) command line options. So i would change Code: ### Starting database dumps for i in `mysql -u$dbuser -p$dbpassword -Bse 'show databases'`; do log "Starting mysqldump $i" `mysqldump -u$dbuser -p$dbpassword $i --allow-keywords --comments=false --add-drop-table > $tmpdir/db-$i-$FDATE.sql` $TAR $ARG $BACKUPDIR/$MDATE/db-$i-$FDATE.tar.bz2 -C $tmpdir db-$i-$FDATE.sql rm -rf $tmpdir/db-$i-$FDATE.sql log "Dump OK. $i database saved OK!" done to Code: ### Starting database dumps for i in `mysql -u$dbuser -p$dbpassword -Bse 'show databases'`; do log "Starting mysqldump $i" `mysqldump -u$dbuser -p$dbpassword $i --allow-keywords --comments=false --add-drop-table --hex-blob --quote-names > $tmpdir/db-$i-$FDATE.sql` $TAR $ARG $BACKUPDIR/$MDATE/db-$i-$FDATE.tar.bz2 -C $tmpdir db-$i-$FDATE.sql rm -rf $tmpdir/db-$i-$FDATE.sql log "Dump OK. $i database saved OK!" done
I had a closer look at how the incremental directory backup is done here. Code: # If it's not the first day of the month we make incremental backup if [ ! -e $tmpdir/full-backup$XX.lck ] ; then log "Starting daily backup for: $YX" NEWER="--newer $FDATE" $TAR $NEWER $ARG $BACKUPDIR/$MDATE/i$XX-$FDATE.tar.bz2 $YX -X $tmpdir/excluded log "Daily backup for $YX done." else log "Lock file for $YX full backup exists!" fi I think this could lead to problems if (for any reason) one incremental run did not work. You could be missing files in the archives then, as the --newer flag is date-based. Wouldn't it be a good idea to use the incremental flag of tar? So instead of using Code: $TAR $NEWER $ARG $BACKUPDIR/$MDATE/i$XX-$FDATE.tar.bz2 $YX -X $tmpdir/excluded you could use Code: $TAR $ARG -g $BACKUPDIR/$MDATE/backup$XX.state $BACKUPDIR/$MDATE/i$XX-$FDATE.tar.bz2 $YX -X $tmpdir/excluded Of course you would have to add the -g $BACKUPDIR/$MDATE/backup$XX.state to the full backup command, too.
tar minuses Hello, In my case, the tar incremental backup is not a very good solution. In the case you loose a incremental backup with my script you can restore the others. See below what's happening with tar incremental backups. With tar "if you plan to create more ‘level 1’ backups, it is necessary to create a working copy of the snapshot file before running tar". With the script this is not the case. The second minus from tar is "Incremental dumps depend crucially on time stamps, so the results are unreliable if you modify a file's time stamps during dumping". My script does not suffer from this and it was created partially because of this tar problem. You can read here more: http://www.gnu.org/software/automake/manual/tar/Incremental-Dumps.html The quoted examples are from the link above.
to be changed Hello, In the next version of the script perhaps I might fix this and some other small issues.
Hello! I have a problem with this line... all is working ok but... in this line I get an error when trying to restore... Code: mysql -u$dbuser -p$dbpassword $rdb < $TAR -xvjp $BACKUPDIR/$YDATE-$MDATE/$i the $TAR variable is ok (echo $TAR shows its value), but, if I run: echo mysql -u$dbuser -p$dbpassword $rdb < $TAR -xvjp $BACKUPDIR/$YDATE-$MDATE/$i I get: mysql -uuser -ppassword notremember < -xvjp /mibackupdir/date/blabla It is like mysql were eating the value of $TAR!!! I write ANOTHER $TAR in the sentence... (... < $TAR $TAR ... ) and the echo shows it correctly: mysql -uuser -ppassword notremember < /TARPATH -xvjp /mibackupdir/date/blabla but I get an error =( I am using Debian, using bash as shell... could you help me? I think that a work around could to untar in a tmp directory, and next do the dump... (this workaround would be great... I didn't have time to modify it...) Thanks for this perfect and usefull script.
? I've have this script working in multiple debian machines, different stable, but never had any issues... Right now I can't imagine what is happening in your particular case, but please if you find out what was wrong, or if you find a solution to your case, or maybe how can I reproduce the error, please write here. By the way what's your Debian version? What packages are installed on your computer? Did you try to duplicate the install onto a virtual machine (virtualbox or vmware) to check if the problem persist? Thank you for your time!
I think cyrus1997 is right. Can you please check. For me in the script there is a dot (.) after /root and this prevents from backing up the /root directory. If I am wrong, please tell me why. working with IspConfig3 on Debian Squeeze 64bit thank you for your work and you time
