ISPConfig 3 - back-res a Backup and Restore script

Few errors while executing the back-res script

Hello,

I'm trying to use the back-res script. I'm getting a few errors and it doesn't work for me. I'm using ISPConfig 3 on Ubuntu.

First I got this:

Code:

mysqldump: Got error: 1044: Access denied for user 'root'@'localhost' to database 'information_schema' when using LOCK TABLES

I checked the rights and the root user has all rights globally, also for LOCK TABLES. I added "--single-transaction" to the mysqldump link in the script. But I find the error message strange because all the necessary rights are provided. I also added LOCK TABLES specifically for the information_schema database but I got the same message.

After adding the single transaction line the script would continue but then I got this message.

Code:

/bin/tar: /root.: Cannot stat: No such file or directory
/bin/tar: Exiting with failure status due to previous errors

Any thoughts? It's a pretty basic ISPConfig setup. Only a few domains for private use. I used the HowtoForge guido for installing ISPConfig 3 on Ubuntu.

 

update

I was able to get the backup-res to restore the directories. I had to mkdir /tmp/tmpbck I am running the script as root. Don't know why it will not create the /tmp directories.
Now when I try to run as root ./backup-res db all 2011-04-01 this is the error I get

date: invalid date `11/'
./backup-res: line 448: [: : integer expression expected

here is line 448
if [ "$RD" -gt "$ID" ] ; then

Restoring all mysql databases from date 2011-04-01 to local server:
mysql: unknown option '-x'
mysql: unknown option '-x'
mysql: unknown option '-x'
mysql: unknown option '-x'
mysql: unknown option '-x'
mysql: unknown option '-x'
mysql: unknown option '-x'
mysql: unknown option '-x'
mysql: unknown option '-x'
mysql: unknown option '-x'
All restore jobs done!
Database all restored to date 2011-04-01!
./backup-res: line 573: /tmp/tmpbck/maildata: No such file or directory
The data bases are not restored.

Thanks so much for you help and patient with me and the others.

 

Some errors

Hello,

- for the first error please check your mysql version, use latest available for Ubuntu. Maybe this can help: http://bugs.mysql.com/bug.php?id=21527
For what I know there is no need to modify the database dumps lines in the script... I will make some checks later this weekend if I'll have some spare time.

- for the second error please check your dirs you want to backup, you have some dot "." in the message and the script complains about not finding the "/root." directory, not the "/root"...

If you have more problems, or you can figure out a way to get rid of the mysql error please return here so your message helps others too.

 

Date problems

Hello,

I think there is a date formatting problem on your system...

Can you run the next commands and post the output here:

Code:

date +%d 
date +%F
date +%Y-%m

The output should be for example:

Code:

user@main:~$ date +%d
15
user@main:~$ date +%F
2011-04-15
user@main:~$ date +%Y-%m
2011-04
gabi@main:~$

The last mysql errors are because of the date problems.

With the tempdir, I don't know why you get the error, there is the "check_tempdir" function which should create the tempdir if it's not there already and it's only failling if the /tmp dir does not exist (it's not a dir). I can put the "-p" option to mkdir but I don't know a linux computer whitout /tmp... so it's like a security option... I'm open to suggestions

 

Posting the date code

Here is the results
root@c3po:~# date +%d
15
root@c3po:~# date +%F
2011-04-15
root@c3po:~# date +%Y-%m
2011-04

Here is the result of the ls -l of /tmp

drwxrwxrwx 6 root root 4096 2011-04-15 14:35 tmp

 

Hello,

The date results seems ok, but your temp dir does not have the sticky bit set (permissions like drwxrwxrwxt). I'm not sure this can influence the script so much...

On a second review of your errors...
- do you have any dir with "-" in the name, excepting the date? This can be a problem because the date of the backup archives is extracted from their names. (Please read the detailed description)
- for the mysql error - do you have a line in your script identical with the one below?

Code:

mysql -u$dbuser -p$dbpassword $rdb < $TAR -xvjp $BACKUPDIR/$YDATE-$MDATE/$i

If you do then the error is because $TAR variable is empty, which means the script can't find tar on your system (which tar)
If you don't have the line identical with the one above please get the latest version of the script.

 

More info on error

I changed the sticky bit this morning
drwxrwxrwt 5 root root 4096 2011-04-17 08:01
still get the error

touch: cannot touch `/tmp/tmpbck/datestart': No such file or directory
touch: cannot touch `/tmp/tmpbck/dateend': No such file or directory
find: `/tmp/tmpbck/datestart': No such file or directory
find: `/tmp/tmpbck/datestart': No such file or directory
find: `/tmp/tmpbck/dateend': No such file or directory

When I tried to do the ./backup-res db all 2011-04-01
I get this
date: invalid date `11/'
./backup-res: line 448: [: : integer expression expected
this is line 448
if [ "$RD" -gt "$ID" ] ; then

I am not sure what the `11 is all about
I will post on some of the other sites about the making and deleting of the /tmp/tmpbck and see what I can find

Here is the line in my backup-res
mysql -u$dbuser -p$dbpassword $rdb < $TAR -xvjp $BACKUPDIR/$YDATE-$MDATE/$i

here is the only part of the script I made changes to and I do have the latest

###############################
### Begin variables section ###
###############################

# Change the variables below to fit your computer/backup

COMPUTER=`cat /etc/hostname | awk 'NR==1{print $1}'` # name of this computer
DIRECTORIES="/var/lib/mysql /var/www
/var/vmail /etc" # directories to backup
EXCLUDED="/bck /tmp /dev /proc /sys /srv /media
/var/adm /var/cache /var/run /var/lock /lib/init/rw /var/tmp
/var/log/verlihub /var/lib/amavis /var/amavis /var/spool/postfix/p*
/var/spool/postfix/var *.pid *.lock *.lck" # exclude those dir's and files
BACKUPDIR="/bck/$COMPUTER" # where to store the backups
dbuser="root" # database user
dbpassword="xxxxxxxxxxxx" # database password
email="[email protected]" # mail for the responsible person
TAR=`which tar` # name and location of tar
ARG="-cjpSPf" #sparse # tar arguments P = removed /.
EARG="-xjpf" # tar extract arguments P = removed /
tmpdir="/tmp/tmpbck" # temp dir for database dump and other stuff
del_en="yes" # Enable delete of files if used space percent > than $maxp (yes or anything else)
maxp="85" # Max percent of used space before start of delete
hm="2359" # last minute of the day = last minute of the restored backup of the day restored

###################################
### End user editable variables ###
###################################

as for the "-" in the file names, in /var/lib/mysql there are not dir or files with that in it they all have "_". In the /var/www folder there are folders with that "-" in it, but those files restored. in the /var/www there are php-cgi-scripts and php-fcgi-scripts that I gues were made by ISPconfig 3 install...

Thanks again for your help...is there anything else you can think of I can get that may help to solve the issue.

 

Dirty fixes

Hello,

a quick fix for the tmpdir is to change the line:

Code:

mkdir $tmpdir

in

Code:

mkdir -p $tmpdir

If the database names contains the "-" or the "_" characters is another thing wich can cause problems. If the date from the filename for the databases is not in form 2011-04-17 there is another point of failure.

But still the error from the 448 line is because $RD it's not an integer. RD should be the first backup date calculated in epoch,

Code:

date --date "$resdate" +%s

So if all of the date parts are ok (previous commands you run), why RD is not? Maybe $redate is not what it should be... And $resdate
is obtained from those lines:

Code:

year=`ls -ctF $BACKUPDIR | grep -v ^log/ | tail -n 1 | cut -d "-" -f 2`
md=`ls -ctF $BACKUPDIR | grep -v ^log/ | tail -n 1 | cut -d "-" -f 3`
day=`ls -ctF $BACKUPDIR | grep -v ^log/ | tail -n 1 | cut -d "-" -f 4 | cut -d "." -f 1`
resdate=$year$md$day

So the problem it's because something strage happens when we extract the date parts from existing files.
I will list here the "debug procedure", hope we'll be able to solve this issue now:
- We can start with the $BACKUPDIR which is composed from the /etc/hostname file.
- If that is correct we can move to the next step,

Code:

ls -ctF $BACKUPDIR

command which you can run in the comand line, like

Code:

ls -ctF /bck/your_hostname

where your_hostname is the one from the /etc/hostname
- If that list seems like:

Code:

2011-04/                     full_var-2011-04-15.tar.bz2  full_sbin-2011-04-15.tar.bz2   full_lib64-2011-04-15.tar.bz2  full_home-2011-04-15.tar.bz2  full_boot-2011-04-15.tar.bz2  log/
full_www-2011-04-15.tar.bz2  full_usr-2011-04-15.tar.bz2  full_root.-2011-04-15.tar.bz2  full_lib-2011-04-15.tar.bz2    full_etc-2011-04-15.tar.bz2   full_bin-2011-04-15.tar.bz2

we can move to the next step:
- the first pipe

Code:

ls -ctF /bck/your_hostname | grep -v ^log/

the result should be like

Code:

2011-04/
full_www-2011-04-15.tar.bz2
full_var-2011-04-15.tar.bz2
full_usr-2011-04-15.tar.bz2
full_sbin-2011-04-15.tar.bz2
full_root.-2011-04-15.tar.bz2
full_lib64-2011-04-15.tar.bz2
full_lib-2011-04-15.tar.bz2
full_home-2011-04-15.tar.bz2
full_etc-2011-04-15.tar.bz2
full_boot-2011-04-15.tar.bz2
full_bin-2011-04-15.tar.bz2

- then we add the next pipe

Code:

ls -ctF /bck/your_hostname  | grep -v ^log/ | tail -n 1

and the result should be the first file, like:

Code:

full_bin-2011-04-15.tar.bz2

- we then proceed to the last pipe,

Code:

ls -ctF /bck/your_hostname  | grep -v ^log/ | tail -n 1 | cut -d "-" -f 2

which should give us the year like

Code:

2011

Hope you can follow those and post where the output is different so we can see where the problem is.

 

Awesome script, really.

I read a post here about some user asking for "remote backup". Ok, it should be possible using sshfs...

______________________________________________________

In debian (using "test" user and remote.domain.com as fqdn of the remote computer as example)

apt-get -y install sshfs
sudo -i
mkdir /localmountdir
adduser test fuse

sshfs [email protected]:/remotefoldertomount /localmountdir -p 22

next, you must specify "/localmountdir/" as backup dir in go0ogl3's script.
_____________________________________________________

I can access the mounted fs with this, but I can't write on it (permission denied)...

Maybe it is not working because I don't allow access in ssh for "root user". I have to make a few tests using root as user to login in ssh... if someone of you could help me with this, maybe we can use this script to make a backup on a remote computer.

Thanks to all,

edit: ok, as root it works perfect (you must add root to fuse group, don't use sudo, login as root in ssh and sshfs, it works for perfect).
I need to make it work using another user, because I don't want to allow root login in ssh/sshfs, please help! )

edit2: I made it work without root access. You must chown the remote folder firts (in the server) and next you must mount it on the "local" server. There is no way to chown files on mounter remote folder, but it is not a problem for this scripts, because you don't need any special owner for the tar.gz files... remember to "restore" the files in a folder OUT of the mounter fs, in order to keep original ownership of the files.

Regards

 

Backp-res can be use for ISPConfig 2 too?

I read the thread and seems that everybody accept that it is a very nice script and working well.
Can be that script used on ISPConfig 2?
I'm looking to have a production server ( ubuntu 10.10 ) and a test server, same configuration with same ISPconfig version and every nigt the script to make a full backup( incremental ) , send it to test server and other script to restore the backup . If anything hapent to the production server I have only to change the ip adreses .
Anybody have that scenario working?

 

I think that it should work, but maybe rsync is better for what you want (but you will not have an incremental backup)

 

Hi, great script.
I would suggest using the --hex-blob option and the field and database name quoting when using mysqldump.

 

ISPC 2 backup and restore

Thank you for suggestion, do you have any HOWTO steps for that. I'm sure that it is many users will like to have an automatic backup and restore module.


Thanks,

 

I just meant the back-res script uses mysqldump without the --hex-blob and --quote-names (-Q) command line options.

So i would change

Code:

    ### Starting database dumps
    for i in `mysql -u$dbuser -p$dbpassword -Bse 'show databases'`; do
	log "Starting mysqldump $i"
	`mysqldump -u$dbuser -p$dbpassword $i --allow-keywords --comments=false --add-drop-table > $tmpdir/db-$i-$FDATE.sql`
	$TAR $ARG $BACKUPDIR/$MDATE/db-$i-$FDATE.tar.bz2 -C $tmpdir db-$i-$FDATE.sql
	rm -rf $tmpdir/db-$i-$FDATE.sql
	log "Dump OK. $i database saved OK!"
    done

to

Code:

    ### Starting database dumps
    for i in `mysql -u$dbuser -p$dbpassword -Bse 'show databases'`; do
	log "Starting mysqldump $i"
	`mysqldump -u$dbuser -p$dbpassword $i --allow-keywords --comments=false --add-drop-table --hex-blob --quote-names > $tmpdir/db-$i-$FDATE.sql`
	$TAR $ARG $BACKUPDIR/$MDATE/db-$i-$FDATE.tar.bz2 -C $tmpdir db-$i-$FDATE.sql
	rm -rf $tmpdir/db-$i-$FDATE.sql
	log "Dump OK. $i database saved OK!"
    done

 

I had a closer look at how the incremental directory backup is done here.

Code:

	    # If it's not the first day of the month we make incremental backup
	    if [ ! -e $tmpdir/full-backup$XX.lck ] ; then
		log "Starting daily backup for: $YX"
		NEWER="--newer $FDATE"
		$TAR $NEWER $ARG $BACKUPDIR/$MDATE/i$XX-$FDATE.tar.bz2 $YX -X $tmpdir/excluded
		log "Daily backup for $YX done."
	    else
	    log "Lock file for $YX full backup exists!"
	    fi

I think this could lead to problems if (for any reason) one incremental run did not work. You could be missing files in the archives then, as the --newer flag is date-based.

Wouldn't it be a good idea to use the incremental flag of tar?

So instead of using

Code:

$TAR $NEWER $ARG $BACKUPDIR/$MDATE/i$XX-$FDATE.tar.bz2 $YX -X $tmpdir/excluded

you could use

Code:

$TAR $ARG -g $BACKUPDIR/$MDATE/backup$XX.state $BACKUPDIR/$MDATE/i$XX-$FDATE.tar.bz2 $YX -X $tmpdir/excluded

Of course you would have to add the -g $BACKUPDIR/$MDATE/backup$XX.state to the full backup command, too.

 

tar minuses

Hello,

In my case, the tar incremental backup is not a very good solution. In the case you loose a incremental backup with my script you can restore the others. See below what's happening with tar incremental backups.

With tar "if you plan to create more ‘level 1’ backups, it is necessary to create a working copy of the snapshot file before running tar". With the script this is not the case.
The second minus from tar is "Incremental dumps depend crucially on time stamps, so the results are unreliable if you modify a file's time stamps during dumping". My script does not suffer from this and it was created partially because of this tar problem.

You can read here more: http://www.gnu.org/software/automake/manual/tar/Incremental-Dumps.html
The quoted examples are from the link above.

 

to be changed

Hello,

In the next version of the script perhaps I might fix this and some other small issues.

 

Hello!

I have a problem with this line... all is working ok but... in this line I get an error when trying to restore...

Code:

		mysql -u$dbuser -p$dbpassword $rdb < $TAR -xvjp $BACKUPDIR/$YDATE-$MDATE/$i

the $TAR variable is ok (echo $TAR shows its value), but, if I run:

echo mysql -u$dbuser -p$dbpassword $rdb < $TAR -xvjp $BACKUPDIR/$YDATE-$MDATE/$i

I get: mysql -uuser -ppassword notremember < -xvjp /mibackupdir/date/blabla

It is like mysql were eating the value of $TAR!!!

I write ANOTHER $TAR in the sentence... (... < $TAR $TAR ... ) and the echo shows it correctly:

mysql -uuser -ppassword notremember < /TARPATH -xvjp /mibackupdir/date/blabla

but I get an error =(

I am using Debian, using bash as shell... could you help me? I think that a work around could to untar in a tmp directory, and next do the dump... (this workaround would be great... I didn't have time to modify it...)

Thanks for this perfect and usefull script.

 

?

I've have this script working in multiple debian machines, different stable, but never had any issues...
Right now I can't imagine what is happening in your particular case, but please if you find out what was wrong, or if you find a solution to your case, or maybe how can I reproduce the error, please write here.

By the way what's your Debian version? What packages are installed on your computer? Did you try to duplicate the install onto a virtual machine (virtualbox or vmware) to check if the problem persist?

Thank you for your time!

 

I think cyrus1997 is right. Can you please check.
For me in the script there is a dot (.) after /root and this prevents from backing up the /root directory.
If I am wrong, please tell me why.

working with IspConfig3 on Debian Squeeze 64bit
thank you for your work and you time