Mirroring Questions & Info (DRBD+OCFS2 instead of GlusterFS) Ive got 2 identical servers I am trying to do a master-master cluster with. I have mysql running in master-master replication successfully across both servers and using DRBD + OCFS2 I also have a shared filesystem for www/vmail. I installed them both identically, websvs1 & websvs2; and configured websvs2 to be a mirror of websvs1 in ISPConfig3. My first test was to add a client & site, websvs1 created the directory structure and its visible from websvs2.. however the system users accounts were not created on websvs2 so the UID/GID of the files are unknown from second server... did I mess something up or is this expected? Should i just start syncing the passwd/shadow/group files using something else? (csync2 perhaps). I kinda expected the system user accounts to mirror each-other. Regards, Ryan
Please ensure that the mysql master-master replication excludes the "mysql" and the ispconfig databases on both servers. ISPConfig is syncing the contents of these databases internally and ispconfig will fail if these databases are mirrored otherwise.
The mysql table is not replicated and I have both servers configured to use seperate databases (dbispconfig1 & dbispconfig2).. I even have a 3rd server mirroring the first that I am hoping to use for shells/ftp thats not a member of the mysql-cluster and gets its filesys via NFS off the cluster.. That server is configured pretty close to the first two and runs in a VM, its not getting any updates to the passwd/shadow/group files so far.. I havent done much testing past simply creating a user and site, I had no problems mirroring DNS services on external servers when I tested it.. Only thing in the cron-logs on websvs2 is: Code: PHP Warning: Invalid argument supplied for foreach() in /usr/local/ispconfig/server/mods-available/monitor_core_module.inc.php on line 1118 /bin/sh: /usr/local/bin/run-getmail.sh: Permission denied repquota: Operation not supported for filesystems with hidden quota files! repquota: Operation not supported for filesystems with hidden quota files!
Seems as if the getmail user ahs no permissions to run that script on your server. There must be a config problem with your Linux file system quota. Please check with: repquota -avug Regarding the config replication problem, most likely there is a problem with mysql permissions, so that the slave can not connect to the mysql master database. Please enable loglevel debug for the salve server in ispconfig master controlpanel and then run this command on the slave as root user and post the output: /usr/local/ispconfig/server/server.sh
I fixed the getmail.sh, was being denied by Trusted Path Execution (TPE) in GrSecurity, had to tweak permissions but its working now. The quota error is apparently an issue with OCFS2, it does not support repquota.. I dunno what to do bout it other than look for a upstream fix of somesort. I'll check out debug logs here shortly and see if that sheds any light.
websvs1: websvs2: Code: 26.08.2011-15:19 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 26.08.2011-15:19 - DEBUG - Found 8 changes, starting update process. 26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'. 26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'. 26.08.2011-15:19 - WARNING - Network configuration disabled in server settings. 26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'. 26.08.2011-15:19 - DEBUG - Processed datalog_id 21 26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'apps_vhost_plugin' raised by event 'server_update'. 26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'network_settings_plugin' raised by event 'server_update'. 26.08.2011-15:19 - WARNING - Network configuration disabled in server settings. 26.08.2011-15:19 - DEBUG - Calling function 'update' from plugin 'postfix_server_plugin' raised by event 'server_update'. 26.08.2011-15:19 - DEBUG - Processed datalog_id 22 26.08.2011-15:19 - DEBUG - Calling function 'restartHttpd' from module 'web_module'. 26.08.2011-15:19 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 26.08.2011-15:20 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 26.08.2011-15:20 - DEBUG - Found 4 changes, starting update process. 26.08.2011-15:20 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 26.08.2011-15:21 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock 26.08.2011-15:21 - DEBUG - Found 4 changes, starting update process. 26.08.2011-15:21 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
so I must have messed something up, uninstalled and cleaned up the databases then set it all up again... seems to be working so far thanks. BTW DRBD+OCFS2 is performing way better than GlusterFS in my tests, plus it supports all kernel platforms.. not just x86. (Sparc64 is what Im running on).. The problem may be in getting Quota's to work properly... if your setting up a Primary/Secondary setup then you can use DRBD+ext3/ext4 and it'll probably work just fine... OCFS2 lets both servers write to the same FS *edit* upstream repquota works with OCFS2, uninstalled quotatool in debian and built the latest version off sourceforge and quotas work just fine.
I've considered this setup as well for the next version of the guide as I was not satisfied with the glusterfs performance. May you post the commands and config files used in your setup so that we can include as option it in the multiserver guide?
Sure, here are the issues ive found using Debian Squeeze. Upgrade Kernel your self, I had issues w/OCFS2 in the 2.6.32 version so I updated kernel to latest version and its been smooth.. (After a while quotas stoped working and I had issues setting permissions on files, I upgraded kernel and reformatted and it hasent happened again.. yet) Take this time to install the GrSecurity patch, configure it for "high security" and disable/dont use TPE as it'll deny a few of ISPConfig3's scripts.. The GrSecurity patch will greatly harden your system, it comes with alot of additional security to chroots... use make-kpkg so it wraps your kernel into a .deb after rebooting into new kernel I built the latest DRBD module against my kernel using: http://oss.linbit.com/drbd/8.4/drbd-latest.tar.gz uninstall quotatools debian package, download and build the latest off SF: http://sourceforge.net/projects/linuxquota/files/quota-tools/ (I installed 4.0) Code: apt-get remove quotatools apt-get builddep quotatools Link your servers together directly, you dont want your servers becoming split-brain if a switch goes down. You should be using Gigabit Ethernet and they will auto-crossover with a straight-thru cable.. In my case bonded 3 Gigabit connections and enabled jumbo-frames: /etc/network/interfaces Configure your hosts file on each server so they will talk through the direct-link. Fdisk drives on both servers so they have a common size, also create a partition for DRBD Meta-Disk. In my case I made a 1GB Meta Partition and a 1460GB File Partition on both servers. (if you can put the meta partition on separate disk for best performance) create a /etc/drbd.d/r0.res file: Notice the startup section is commented out, you will want to uncomment this once everything is up and running.. but until then you dont want dual primaries until after we initialize DRBD.. (note starting it will error the first time, we just want to load modules) primary: Code: /etc/init.d/drbd start drbdadm create-md r0 drbdadm connect r0 drbdadm primary r0 on the slave: Code: /etc/init.d/drbd start drbdadm create-md r0 drbdadm -- --clear-bitmap new-current-uuid r0 drbdadm connect r0 drbdadm primary r0 This should start up DRBD and skip the inital sync.. /etc/init.d/drbd status Code: drbd driver loaded OK; device status: version: 8.4.0 (api:1/proto:86-100) GIT-hash: 28753f559ab51b549d16bcf487fe625d5919c49c build by root@websvs1, 2011-08-23 02:44:22 m:res cs ro ds p mounted fstype 0:r0 Connected Primary/Primary UpToDate/UpToDate C Now for OCFS2, configure backports on both servers /etc/apt/sources.list Code: deb http://backports.debian.org/debian-backports squeeze-backports main and install ocfs2-tools Code: apt-get update;apt-get install -t squeeze-backports ocfs2-tools Configure OCFS2 cluster on both servers /etc/ocfs2/cluster.conf Code: node: ip_port = 7777 ip_address = 192.168.253.1 number = 0 name = websvs1 cluster = websvs node: ip_port = 7777 ip_address = 192.168.253.2 number = 1 name = websvs2 cluster = websvs cluster: node_count = 2 name = websvs Then start OCFS2 with: Code: /etc/init.d/o2cb restart on the primary lets format the DRBD raid using OCFS2: and then you should be able to mount it on both servers: Code: /sbin/mount.ocfs2 /dev/drbd0 -o usrquota,grpquota /srv check ocfs2 status: Code: /etc/init.d/o2cb status Driver for "configfs": Loaded Filesystem "configfs": Mounted Stack glue driver: Loaded Stack plugin "o2cb": Loaded Driver for "ocfs2_dlmfs": Loaded Filesystem "ocfs2_dlmfs": Mounted Checking O2CB cluster websvs: Online Heartbeat dead threshold = 31 Network idle timeout: 30000 Network keepalive delay: 2000 Network reconnect delay: 2000 Checking O2CB heartbeat: Active then simply move /var/www to /srv and link it back, do the same for /var/vmail like the guide says too. Now dont add /srv to fstab like you may be tempted, debain will try to fsck it at boot which will fail and force you to interact to get the system to boot.. I added a few lines to /etc/rc.local to set up the FileSystems. Code: /etc/init.d/ocfs2 start /sbin/mount.ocfs2 /dev/drbd0 -o usrquota,grpquota /srv /etc/init.d/apache2 start exit 0 Think that should cover it, I'll throw it into production soon and let you know if there is anything else that needs addressed. Using DRBD+OCFS2 on 2 identical disks gives nearly native read speeds and about half the write speeds of a native disk.. uses very little resources compared to GlusterFS and the performance is leaps and bounds better.. I dont recommend putting your DB on it tho, use MySQL Replication for that and put your DB's on a local raid-0/10 or SSD if you can. additional thoughts, if your going to be processing a ton of mail it might be beneficial performance wise to create a separate drbd resource for vmail storage and format it using a smaller block size.. now would be the perfect time to set that up. feature request: It'd be great if ISPConfig3 monitored the DRBD & OCFS status like it does soft-raids.. Cheers, -R