ISPConfig3 Backup and RSync

30uke · May 29, 2024

I wrote two small scripts to dump and sync databases, email folders, and other specific directories. These scripts are useful for restoring an ISPConfig server from bz2 archives.
I hope these scripts will be helpful to others, so I'm sharing them here:
https://ict-diensten.com/pub/scripts/ispconfig-backup-and-rsync.sh.tar.bz2

I have stored these scripts in '/opt/scripts' and execute them with cron. Please note the following:

Set the permissions of the '/root/.secrets/rsync-nas' and '/root/.my.cnf' files to 600 for security.
If you do not use ISPConfig's backup feature for websites, include '/var/www' in the otherdirs variable.

Feel free to use and adapt these scripts to fit your needs!

backup.sh

Code:

#!/bin/bash
###############################################################
# Backup Script
# Author: Bouke Jasper Henstra, bouke -at- ict-diensten.com
#
# Functionality:
# 1. Dump databases
# 2. Dump mail directories
# 3. Dump specified directories
#
# Does not include:
#    Dir '/var/www' Reason: this should be done via ISPConfig
#
# Standard dump dir is
#    Dir '/var/backup' (tip: do not change)
#
# Dependencies:
# - File: /root/.my.cnf
#     [client]
#     user=root
#     password=password
#
###############################################################

###############################################################
# Vars
###############################################################
# Retention
retention_days=30  # Number of days to keep old backups
# Define the number of days to retain logs
retain_logs_days=30
# Timestamp
tstamp=$(date +%Y%m%d_%H%M%S)
# Logging: dirs
parentlogdir=/var/log/backups
logdir=$parentlogdir/$tstamp
logfile=$logdir/backup.log
# MySQL
sqlconfig=/root/.my.cnf  # Make sure to have [client] section with user and password
# SQLDir, destination
sqldir=/var/backup/mysql
# Mail, source
maildir=/var/vmail
# Mail, destination
backupmaildir=/var/backup/mail
# Other directories, destination
backupdir=/var/backup/other
# Other directories to backup, source
otherdirs=("/etc" "/root/.acme.sh" "/usr/local/ispconfig" "/opt/scripts" "/var/spool/cron" "/var/spool/postfix")

###############################################################
# Pre Tasks
###############################################################
# Create dirs
mkdir -p $logdir
mkdir -p $sqldir
mkdir -p $sqldir/$tstamp
mkdir -p $backupmaildir
mkdir -p $backupmaildir/$tstamp
mkdir -p $backupdir
mkdir -p $backupdir/$tstamp

# Logging
exec > >(tee -a ${logfile} )
exec 2> >(tee -a ${logfile} >&2)

###############################################################
# Run mysqldump
###############################################################
echo "*** Starting database dump at $tstamp"
DATABASES=$(/usr/bin/mysql --defaults-extra-file=$sqlconfig -Bse 'show databases')

for db in ${DATABASES[@]}
do
  if [[ "$db" != "performance_schema" && "$db" != "mysql" && "$db" != "information_schema" && "$db" != "sys" ]]; then
    dumpfile=$sqldir/$tstamp/mysqldump-${db}-$tstamp.sql.bz2
    echo "Dumping database $db to $dumpfile"
    if /usr/bin/mysqldump --defaults-extra-file=$sqlconfig $db --single-transaction -R | bzip2 -c > $dumpfile; then
      echo "Successfully dumped $db"
    else
      echo "Failed to dump $db" >&2
    fi
  fi
done

###############################################################
# Backup mail directories
###############################################################
echo "*** Starting mail directories backup at $tstamp"
tarfile=$backupmaildir/$tstamp/mail-backup-$tstamp.tar.bz2

# Pause mail services here (optionally)
systemctl stop postfix
systemctl stop dovecot

if tar -cjf $tarfile -C $maildir .; then
  echo "Successfully backed up mail directories to $tarfile"
else
  echo "Failed to back up mail directories" >&2
fi

# Resume mail services here (optionally)
systemctl start postfix
systemctl start dovecot

# Calculate and store hash for integrity check
md5sum $tarfile > $tarfile.md5

###############################################################
# Backup other specified directories
###############################################################
echo "*** Starting backup of other directories at $tstamp"

for dir in "${otherdirs[@]}"
do
  dirname=$(basename $dir)
  tarfile=$backupdir/$tstamp/${dirname}-backup-$tstamp.tar.bz2
  echo "Backing up $dir to $tarfile"

  # Stop Postfix if directory is /var/spool/postfix
  if [[ "$dir" == "/var/spool/postfix" ]]; then
    systemctl stop postfix
  fi

  if tar -cjf $tarfile -C $(dirname $dir) $dirname; then
    echo "Successfully backed up $dir to $tarfile"
  else
    echo "Failed to back up $dir" >&2
  fi

  # Start Postfix if directory is /var/spool/postfix
  if [[ "$dir" == "/var/spool/postfix" ]]; then
    systemctl start postfix
  fi

  # Calculate and store hash for integrity check
  md5sum $tarfile > $tarfile.md5
done

# Check the status of Postfix and write only the core information to the log file
postfix_status=$(systemctl is-active postfix)
echo "*** Postfix Status: $postfix_status" >> $logfile

# Check the status of Dovecot and write only the core information to the log file
dovecot_status=$(systemctl is-active dovecot)
echo "*** Dovecot Status: $dovecot_status" >> $logfile

# Mail log
mail -s "Backup log s1.xyz.int" -r "root s1.xyz.int <[email protected]>" [email protected] < $logfile

# Remove old backups
find $sqldir -type d -mtime +$retention_days -exec rm -rf {} \;
find $backupmaildir -type d -mtime +$retention_days -exec rm -rf {} \;
find $backupdir -type d -mtime +$retention_days -exec rm -rf {} \;

# Remove old log files and directories in /var/log/backups recursively
find /var/log/backups -mindepth 1 -type d -mtime +$retain_logs_days -exec rm -rf {} +

echo "Backup completed at $(date +%Y%m%d_%H%M%S)"

rsync-backup.sh

Code:

#!/bin/bash

###############################################################
# Backup Sync Script
# Author: Bouke Jasper Henstra, bouke -at- ict-diensten.com
#
# Functionality:
# 1. Sync /usr/backup to remote server via SSH
# 2. Send email with the status
#
# Dependencies:
# - sshpass (apt install sshpass)
# - file '/root/.secrets/rsync-nas'
#     username=username
#     password=password
###############################################################

# Variables
SRC_DIR="/var/backup"
DEST_HOST="1.2.3.4"
DEST_PORT="2222"
DEST_DIR="/volume1/bup-server-s1/s1-bup"
LOG_FILE="/var/log/backup_sync.log"
EMAIL_FROM="root s1.xyz.int <[email protected]>"
EMAIL_SUBJECT="Backup sync status s1.xyz.int"
EMAIL_RECIPIENTS="[email protected]"
CRED_FILE="/root/.secrets/rsync-nas"

# Read credentials
source $CRED_FILE
DEST_USER=$username
DEST_PASS=$password

# Logging
exec > >(tee -a ${LOG_FILE} )
exec 2> >(tee -a ${LOG_FILE} >&2)

# Rsync Command
  RSYNC_CMD="sshpass -p '$DEST_PASS' rsync -avz --no-perms --no-owner --no-group -e 'ssh -p $DEST_PORT' $SRC_DIR/ $DEST_USER@$DEST_HOST:$DEST_DIR"

echo "Starting rsync..."
#echo "Executing: $RSYNC_CMD"  # Debugging line to show the exact command being executed

if eval $RSYNC_CMD; then
  echo "Successfully synced $SRC_DIR to $DEST_USER@$DEST_HOST:$DEST_DIR"
  mail -s "$EMAIL_SUBJECT: Success" -r "$EMAIL_FROM" $EMAIL_RECIPIENTS <<< "Successfully synced /usr/backup to $DEST_USER@$DEST_HOST:$DEST_DIR"
else
  echo "Failed to sync /usr/backup" >&2
  mail -s "$EMAIL_SUBJECT: Failed"  -r "$EMAIL_FROM" $EMAIL_RECIPIENTS <<< "Failed to sync /usr/backup to $DEST_USER@$DEST_HOST:$DEST_DIR"
fi

echo "Backup sync script completed."

Taleman · May 30, 2024

Is there a reason to use sshpass instead of using ssh keys?

30uke · May 30, 2024

Not really. I just hadn't set up SSH keys on my NAS yet. It should work after setting up the SSH key pairs and the following edits:

Code:

#!/bin/bash
###############################################################
# Backup Sync Script
# Author: Bouke Jasper Henstra, bouke -at- ict-diensten.com
#
# Functionality:
# 1. Sync /usr/backup to remote server via SSH (with SHH Key pairs authentication)
# 2. Send email with the status
#
# Dependencies:
# - file '/root/.secrets/rsync-nas'
#     username=username
###############################################################

# Variables
SRC_DIR="/var/backup"
DEST_HOST="1.2.3.4"
DEST_PORT="2222"
DEST_DIR="/volume1/bup-server-s1/s1-bup"
LOG_FILE="/var/log/backup_sync.log"
EMAIL_FROM="root s1.xyz.int <[email protected]>"
EMAIL_SUBJECT="Backup sync status s1.xyz.int"
EMAIL_RECIPIENTS="[email protected]"
CRED_FILE="/root/.secrets/rsync-nas"
SSH_KEY="/root/.ssh/id_rsa"

# Read username from credentials file
source $CRED_FILE
DEST_USER=$username

# Logging
exec > >(tee -a ${LOG_FILE} )
exec 2> >(tee -a ${LOG_FILE} >&2)

# Rsync Command
RSYNC_CMD="rsync -avz --no-perms --no-owner --no-group -e 'ssh -i $SSH_KEY -p $DEST_PORT' $SRC_DIR/ $DEST_USER@$DEST_HOST:$DEST_DIR"

echo "Starting rsync..."
#echo "Executing: $RSYNC_CMD"  # Debugging line to show the exact command being executed

if eval $RSYNC_CMD; then
  echo "Successfully synced $SRC_DIR to $DEST_USER@$DEST_HOST:$DEST_DIR"
  mail -s "$EMAIL_SUBJECT: Success" -r "$EMAIL_FROM" $EMAIL_RECIPIENTS <<< "Successfully synced /usr/backup to $DEST_USER@$DEST_HOST:$DEST_DIR"
else
  echo "Failed to sync /usr/backup" >&2
  mail -s "$EMAIL_SUBJECT: Failed"  -r "$EMAIL_FROM" $EMAIL_RECIPIENTS <<< "Failed to sync /usr/backup to $DEST_USER@$DEST_HOST:$DEST_DIR"
fi

echo "Backup sync script completed."

Log in or Sign up

ISPConfig3 Backup and RSync

30uke Active Member HowtoForge Supporter

Taleman Well-Known Member HowtoForge Supporter

30uke Active Member HowtoForge Supporter

Share This Page

Log in or Sign up

ISPConfig3 Backup and RSync

30uke Active Member HowtoForge Supporter

Taleman Well-Known Member HowtoForge Supporter

30uke Active Member HowtoForge Supporter

Share This Page

Useful Searches