Mirroring Questions

Mirroring Questions & Info (DRBD+OCFS2 instead of GlusterFS)

Ive got 2 identical servers I am trying to do a master-master cluster with.

I have mysql running in master-master replication successfully across both servers and using DRBD + OCFS2 I also have a shared filesystem for www/vmail.

I installed them both identically, websvs1 & websvs2; and configured websvs2 to be a mirror of websvs1 in ISPConfig3.

My first test was to add a client & site, websvs1 created the directory structure and its visible from websvs2.. however the system users accounts were not created on websvs2 so the UID/GID of the files are unknown from second server... did I mess something up or is this expected? Should i just start syncing the passwd/shadow/group files using something else? (csync2 perhaps). I kinda expected the system user accounts to mirror each-other.

Regards,
Ryan

 

The mysql table is not replicated and I have both servers configured to use seperate databases (dbispconfig1 & dbispconfig2)..

I even have a 3rd server mirroring the first that I am hoping to use for shells/ftp thats not a member of the mysql-cluster and gets its filesys via NFS off the cluster.. That server is configured pretty close to the first two and runs in a VM, its not getting any updates to the passwd/shadow/group files so far..

I havent done much testing past simply creating a user and site, I had no problems mirroring DNS services on external servers when I tested it..

Only thing in the cron-logs on websvs2 is:

Code:

PHP Warning: Invalid argument supplied for foreach() in /usr/local/ispconfig/server/mods-available/monitor_core_module.inc.php on line 1118
/bin/sh: /usr/local/bin/run-getmail.sh: Permission denied
repquota: Operation not supported for filesystems with hidden quota files!
repquota: Operation not supported for filesystems with hidden quota files!

 

I fixed the getmail.sh, was being denied by Trusted Path Execution (TPE) in GrSecurity, had to tweak permissions but its working now.

The quota error is apparently an issue with OCFS2, it does not support repquota.. I dunno what to do bout it other than look for a upstream fix of somesort.

I'll check out debug logs here shortly and see if that sheds any light.

 

websvs1:

websvs2:

Code:

26.08.2011-15:19 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
26.08.2011-15:19 - DEBUG - Found 8 changes, starting update process.
26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
26.08.2011-15:19 - WARNING - Network configuration disabled in server settings.
26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'.
26.08.2011-15:19 - DEBUG - Processed datalog_id 21
26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'.
26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'.
26.08.2011-15:19 - WARNING - Network configuration disabled in server settings.
26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'.
26.08.2011-15:19 - DEBUG - Processed datalog_id 22
26.08.2011-15:19 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
26.08.2011-15:19 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
26.08.2011-15:20 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
26.08.2011-15:20 - DEBUG - Found 4 changes, starting update process.
26.08.2011-15:20 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
26.08.2011-15:21 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
26.08.2011-15:21 - DEBUG - Found 4 changes, starting update process.
26.08.2011-15:21 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock

 

so I must have messed something up, uninstalled and cleaned up the databases then set it all up again... seems to be working so far thanks.

BTW DRBD+OCFS2 is performing way better than GlusterFS in my tests, plus it supports all kernel platforms.. not just x86. (Sparc64 is what Im running on).. The problem may be in getting Quota's to work properly... if your setting up a Primary/Secondary setup then you can use DRBD+ext3/ext4 and it'll probably work just fine... OCFS2 lets both servers write to the same FS

*edit*
upstream repquota works with OCFS2, uninstalled quotatool in debian and built the latest version off sourceforge and quotas work just fine.

 

Sure, here are the issues ive found using Debian Squeeze.

Upgrade Kernel your self, I had issues w/OCFS2 in the 2.6.32 version so I updated kernel to latest version and its been smooth.. (After a while quotas stoped working and I had issues setting permissions on files, I upgraded kernel and reformatted and it hasent happened again.. yet) Take this time to install the GrSecurity patch, configure it for "high security" and disable/dont use TPE as it'll deny a few of ISPConfig3's scripts.. The GrSecurity patch will greatly harden your system, it comes with alot of additional security to chroots... use make-kpkg so it wraps your kernel into a .deb

after rebooting into new kernel I built the latest DRBD module against my kernel using:
http://oss.linbit.com/drbd/8.4/drbd-latest.tar.gz

uninstall quotatools debian package, download and build the latest off SF: http://sourceforge.net/projects/linuxquota/files/quota-tools/ (I installed 4.0)

Code:

apt-get remove quotatools
apt-get builddep quotatools

Link your servers together directly, you dont want your servers becoming split-brain if a switch goes down. You should be using Gigabit Ethernet and they will auto-crossover with a straight-thru cable.. In my case bonded 3 Gigabit connections and enabled jumbo-frames:

/etc/network/interfaces

Configure your hosts file on each server so they will talk through the direct-link.

Fdisk drives on both servers so they have a common size, also create a partition for DRBD Meta-Disk. In my case I made a 1GB Meta Partition and a 1460GB File Partition on both servers. (if you can put the meta partition on separate disk for best performance)

create a /etc/drbd.d/r0.res file:

Notice the startup section is commented out, you will want to uncomment this once everything is up and running.. but until then you dont want dual primaries until after we initialize DRBD.. (note starting it will error the first time, we just want to load modules)

primary:

Code:

/etc/init.d/drbd start
drbdadm create-md r0
drbdadm connect r0
drbdadm primary r0

on the slave:

Code:

/etc/init.d/drbd start
drbdadm create-md r0
drbdadm -- --clear-bitmap new-current-uuid r0 
drbdadm connect r0
drbdadm primary r0

This should start up DRBD and skip the inital sync..
/etc/init.d/drbd status

Code:

drbd driver loaded OK; device status:
version: 8.4.0 (api:1/proto:86-100)
GIT-hash: 28753f559ab51b549d16bcf487fe625d5919c49c build by root@websvs1, 2011-08-23 02:44:22
m:res  cs         ro               ds                 p  mounted  fstype
0:r0   Connected  Primary/Primary  UpToDate/UpToDate  C

Now for OCFS2, configure backports on both servers
/etc/apt/sources.list

Code:

deb http://backports.debian.org/debian-backports squeeze-backports main

and install ocfs2-tools

Code:

apt-get update;apt-get install -t squeeze-backports ocfs2-tools

Configure OCFS2 cluster on both servers
/etc/ocfs2/cluster.conf

Code:

node:
        ip_port = 7777
        ip_address = 192.168.253.1
        number = 0
        name = websvs1
        cluster = websvs

node:
        ip_port = 7777
        ip_address = 192.168.253.2
        number = 1
        name = websvs2
        cluster = websvs

cluster:
        node_count = 2
        name = websvs

Then start OCFS2 with:

Code:

/etc/init.d/o2cb restart

on the primary lets format the DRBD raid using OCFS2:

and then you should be able to mount it on both servers:

Code:

/sbin/mount.ocfs2 /dev/drbd0 -o usrquota,grpquota /srv

check ocfs2 status:

Code:

/etc/init.d/o2cb status
Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Stack glue driver: Loaded
Stack plugin "o2cb": Loaded
Driver for "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster websvs: Online
Heartbeat dead threshold = 31
  Network idle timeout: 30000
  Network keepalive delay: 2000
  Network reconnect delay: 2000
Checking O2CB heartbeat: Active

then simply move /var/www to /srv and link it back, do the same for /var/vmail like the guide says too.

Now dont add /srv to fstab like you may be tempted, debain will try to fsck it at boot which will fail and force you to interact to get the system to boot..

I added a few lines to /etc/rc.local to set up the FileSystems.

Code:

/etc/init.d/ocfs2 start
/sbin/mount.ocfs2 /dev/drbd0 -o usrquota,grpquota /srv
/etc/init.d/apache2 start
exit 0

Think that should cover it, I'll throw it into production soon and let you know if there is anything else that needs addressed.

Using DRBD+OCFS2 on 2 identical disks gives nearly native read speeds and about half the write speeds of a native disk.. uses very little resources compared to GlusterFS and the performance is leaps and bounds better.. I dont recommend putting your DB on it tho, use MySQL Replication for that and put your DB's on a local raid-0/10 or SSD if you can.

additional thoughts, if your going to be processing a ton of mail it might be beneficial performance wise to create a separate drbd resource for vmail storage and format it using a smaller block size.. now would be the perfect time to set that up.

feature request: It'd be great if ISPConfig3 monitored the DRBD & OCFS status like it does soft-raids..

Cheers,
-R