Building Ubuntu Server 12.04.5 - FileServer/RAID1 + BackUp

Hi friends,


What & why:
I'm building this fileserver using Ubuntu Server 12.04.5 with RAID1 to make my files accessible, should the worse thing happen. I succeeded setting up Ubuntu Software RAID1 and it works just great (I think).
But, I'm not quite satisfied yet! I want to build a system, where a failed diskdrive will not stop the access to the file's as well as making backups. I'm using 2U cases for all my servers. There's only two 5.25" bays in the front of these 2U cases, and I want the disks in this system to be easily replaced in case of a hardware defect. I'll show the case in the end of the post.
So, what I'm asking all you for, is an idea how to make this FileServer with + BackUp. Which type of disks to use, SSD or ordinary harddrives. How to plcae what and where!?!?

What do I need?
1. I need to be able to change faulty diskdrives in case of a crash.
2. I need to have diskspace for 10 copies/versions of backup. The 11'th backup will overwrite the oldest.

My latest idea is to build it using:
2 x ICY BOX Trayless Mobile Rack 3.5" SATA HDD (quick removal of faulty disk)
2 x 2 SANDisk SSD 240Gb
2 x 2 Western Digital 2Tb harddrives

Install the ICY BOX Trayless Mobile Racks in the two available bays for easy front handling.
Install the two 240Gb SANDisk SSD's in the Mobile Racks above, and setup RAID1 with swapfile and ext4 on the rest of the space.
Install the two 2Tb harddrives from Western Digital internally, and then configurate these for backup.

Issues & problems that needs to be solved:
1. When a disk in Ubuntu Software RAID1 chrashes, you can't see the damage before rebooting the system. Using the known commands to check the array's/md's there's no indication of a problem before the systems is rebooted! That's what I noticed so far.
2. How to set-up the server to be able to send the warning emails the "RAID Monitor Deamon" creates, when a array gets degraded. (related threads 2)
3. How to set up the system/e.g. a shell script to make backup on to the two 2Tb harddrives. 10 copies before overwriting the first. I'm thinking that making these two disks run as standard disk/no RAID! But have a script copy the backup from the RAID1 to both these disks. This way we have a backup on 2 normal harddrives, not SSD. Much easier to do recovery on "real" harddrives intead of SSD. And, we've got 2 harddisk with each a copy of the backup.

So, what am I asking!?!?
A. Some guidence. Am I totally lost here? Is this the wrong way to do stuff like this??
B. Suggestions on how to take content from an array, let's call it "MyData", make a backup as a .rar file on to the two "backup disks". A copy on each drive. Daily. A script maybe and CRON!?
C. Echo to the "screen" (I use SSH/Putty from another computer to remotely control all servers), is a array has gone "degraded".
D. * I'm currently working on the email warning (RAID Monitor Deamon) thanks to Srijan! But, if you have a suggestion on this matter, please don't hesitate to let us know
E. A suggestion to another setup. Another kind of RAID and/or another way to backup data.

I've been working as a datarecovery agent for many years, which is the reason I would like the backups to be on a "normal" harddrive. There's so much more you can do right away having "lost data" from a "normal" harddrive than say a SSD drive. This is why I'm so hooked on getting the backup on to a couple of Western Digital disks. WD disks were the kind of harddrive I saw the fewest times. That's my opinion anyway

Related threads:
http://www.howtoforge.com/forums/showthread.php?p=317901#poststop
http://www.howtoforge.com/forums/showthread.php?p=317900#post317900
http://www.howtoforge.com/forums/showthread.php?p=317619#poststop

 

Project continues here...

Hi,

I went ahead with the project and if you have stumbled over this post and need answers related to this, please look here:

http://www.howtoforge.com/forums/showthread.php?p=322598#post322598

Hope you can use it to build your own server or to solve your issues

 

I have a similar setup, but I run 3 drives in Raid1.

However I have an IcyBox with 5 bays. So what I do is use a script that

1) copies parittion layout to newly added disk
2) since I use 4TB drives and have EFI Mainboard, I need that EFI main parition, so I just DD that one
3) Then I expand the Raid1 to 4 active disks and add the newly added drive to it

I do that every month.

Code:

#!/bin/bash


[[ $2 ]] || { ls -al /dev/disk/by-id; echo "You need to specifiy two arguments. Argument 1 is the origin drive from which partition layout and efi boot partition is copie, Argument two is the new/backup drive. E.g. ./script.sh sda sdd to copy from sda to sdd" >&2; exit 1; }


orig="/dev/${1}"
dest="/dev/${2}"


read -p "Clone system from ${orig} to ${dest}. Press Y to continue: "
echo # Just to move to a new line
if [[ ! "${REPLY}" =~ ^[Yy]$ ]]
then
    exit 1
fi


# Make partition layout
sgdisk -R="${dest}" "${orig}"
sgdisk -G "${dest}"


# Clone the EFI Boot partition
#dd if="${orig}1" of="${dest}1"


# Install GRUB to dest
#grub-install "${dest}"


# Shut down VMs
#for vm in {100..104}
#do
#       qm shutdown "${vm}"
#done


# Grow, Clone, Shrink RAID devices
declare -A raidInfo
raidInfo[0]="2"
raidInfo[1]="3"
raidInfo[2]="4"

for i in "${!raidInfo[@]}"
do

        mdadm --grow --force "/dev/md${i}" --raid-devices=4
        mdadm --zero-superblock "${dest}${raidInfo[$i]}"
        mdadm --manage "/dev/md${i}" --add --write-mostly "${dest}${raidInfo[$i]}"
        mdadm --wait "/dev/md${i}"
        sync
done


# Clone the EFI Boot partition
dd if="${orig}1" of="${dest}1"

# Install GRUB to dest
grub-install "${dest}"


for i in "${!raidInfo[@]}"
do
        mdadm "/dev/md${i}" --fail "${dest}${raidInfo[$i]}"
        mdadm "/dev/md${i}" --remove "${dest}${raidInfo[$i]}"
        mdadm --grow "/dev/md${i}" --raid-devices=3
        echo ""
        echo "Finshed syncing /dev/md${i}"
        echo ""
        echo ""
        echo ""
done


# Restart VM, except the CA
#for vm in {100..103}
#do
#       qm start "${vm}"
#done

you probably need sfdisk instead of sgdisk... not sure how the syntax is there... sgdisk is required if you have larger than 2TB drives.


I also do daily backups within each of the VMs (the above just makes a hotcopy of the server). Here's the backup script I use for the ISPC/Mail Server:

backup.sh

Code:

#!/bin/bash

# USER VARIABLES
backupDir="/backup"                                             # Folder for current and old backups
doMysqlBackup="y"                                               # Set y/n if you want to make MySQL Backup also
mysqlUser="root"                                                # MySQL User
mysqlPasswd="..."                             # MySQL Password
mysqlHost="localhost"                                   # MySQL Host
mysqlDataFolder="/var/lib/mysql"                # Folder containing the binary mysql DBs
mysqlBackupDir="/mysql_backup"                  # Folder for dumping .sql files and binary copies
excludes="/root/backup_excludes"        # File containing exludes
days="30"                                                               # After how many days shall the backups be deleted?



##############################################################################
#                                                                            #
#                            BELOW BE DRAGONS                                #
#                                                                            #
##############################################################################



# Check if important variables are set
if [[ -z "${backupDir}" ]]
then
        echo "The 'backupDir' var cannot be empty. Fix and try again."
        exit 1;
fi
if [[ "${doMysqlBackup}" = "y" ]]
then
        if [[ -z "${mysqlBackupDir}" ]]
        then
                echo "The 'mysqlBackupDir' var cannot be empty. Fix and try again."
                exit 1;
        fi
fi



# Create necessary folders
now=$(date '+%Y-%m'-%d_%H-%M)
mkdir -p "${backupDir}/current"
mkdir -p "${backupDir}/old/${now}"
if [[ "${doMysqlBackup}" = "y" ]]
then
        mkdir -p "${mysqlBackupDir}"
fi

# Make MySQL Backup
if [[ "${doMysqlBackup}" = "y" ]]
then
        # Remove existing mysql backup data
        rm -Rf "${mysqlBackupDir}/*"

        # Dump new files
        for i in $(echo 'SHOW DATABASES;' | mysql -u"${mysqlUser}" -p"${mysqlPasswd}" -h"${mysqlHost}" | grep -v '^Database$')
        do
                mysqldump                                                                                                       \
                -u"${mysqlUser}" -p"${mysqlPasswd}" -h"${mysqlHost}"            \
                -Q -c -C --add-drop-table --add-locks --quick --lock-tables     \
                "${i}" > "${mysqlBackupDir}/${i}.sql";
        done

        # Stop and Restart MySQL to make binary copy of the DBs
        /etc/init.dmysql stop
        cp -a "${mysqlDataFolder}/"* "${mysqlBackupDir}/"
        /etc/init.d/mysql start
fi



# If it's the first day of the month, make full backup (to have all data read again)
testMonth=$(date +%d)
if [[ "${testMonth}" = "01" ]]
then
        rm -Rf "${backupDir}/current"
        mkdir -p "${backupDir}/current"
fi




# Run rsync into current
rsync                                                                                           \
        -avzp --delete --delete-excluded                                \
        --exclude-from="${excludes}"                                    \
        --exclude "${backupDir}/"                                               \
        $mysqlExclude                                                                   \
        /                                                                                               \
        "${backupDir}/current" ;


# Update the mtime to reflecht the snapshot time
touch "${backupDir}/current"


# make hardlinke copy  --> this does the magic of creating snapshot-style incremental backups
cp -al "${backupDir}/current/"* "${backupDir}/old/${now}"




# Remove old backups
#for i in "$( find ${backupDir}/old/ -maxdepth 1 -type d -mtime +${days} )"
#do
#       rm -Rf "${i}"
#done
find "${backupDir}/old/" -maxdepth 1 -type d -mtime +"${days}" -exec rm -Rf "{}" \;

exit 0

backup_excludes -> adjust to your own needs

Code:

/backup/
/dev/
/initrd.img
/lost+found/
/media/
/proc/
/run/
/sys/
/tmp/
/var/lib/mysql/
/vmlinuz

Your questions:

1. When a disk in Ubuntu Software RAID1 chrashes, you can't see the damage before rebooting the system. Using the known commands to check the array's/md's there's no indication of a problem before the systems is rebooted! That's what I noticed so far.

Because of that I make monthly new hot copies. So the full raid is read from beginning to end. Hot Copies plus daily backups are fine for me.



2. How to set-up the server to be able to send the warning emails the "RAID Monitor Deamon" creates, when a array gets degraded.

In the /etc/mdadm/mdadm.conf file just add:
MAILADDR [email protected]

And also be sure to enable a mail sending capability. Either setup a full fledge email server or just set it up as smarthost... that should fairly simple with postfix. Smarthost setup means, that you give login credentials for an other mail service, e.g. gmail and it will they send through there.



3. How to set up the system/e.g. a shell script to make backup on to the two 2Tb harddrives. 10 copies before overwriting the first. I'm thinking that making these two disks run as standard disk/no RAID! But have a script copy the backup from the RAID1 to both these disks. This way we have a backup on 2 normal harddrives, not SSD. Much easier to do recovery on "real" harddrives intead of SSD. And, we've got 2 harddisk with each a copy of the backup.

In my example I use hardlinks. hardlinks basically doubles the reference in the "Table of Contents" of the filesystem. So two entries point to the same data. That is space efficient and works well together with rsync. Because when rsync notices, that a file has altered, it will unlink it first and create a new file. That way you can have a full backup for everyday yet not use too much space if the data doesn't change often.

However, bit rot might a problem. That's why I make monthly a hot copy of the system as well as on every first of the month I make a full rewrite of all data.