Thought I'd share how I obtained HA Nirvana with ISPConfig3. Two Identical Servers: - 6 Core PhenomII @ 3.5GHz w/16GB RAM w/4x1TB SAS Striped - Infiniband Dual Port 10GBps PCIe Card / Dual Intel Gbit Ethernet - OS: Proxmox 1.9 PVE On the servers I configured a few LVM slices, Primary-Tank/Secondary-Tank and Websvs. The tank drives store virtual machines, one for each physical server and the websvs is going to be our shared storage between servers. (in my setup websvs is another dedicated storage array so its IO is separate from VM's) I configured 3 DRBD Devices, one for each LVM slice and configured them communicate directly to each other via IP over Infiniband.. I then created another layer of LVM on top and fed these into Proxmox as Shared LVM storage devices, this allows for Live Migration of KVM Servers among other things. In proxmox I created 2 KVM machines (websvs1 & websvs2) with PVE Storage & Network.. one tied to each physical server. On websvs1 I used the Primary-Tank and on websvs2 I used the Secondary-Tank. I used separate volumes for each physical server even though they are mirrored, this will make recovering from a DRBD spilt-brain scenario easier when you know what data was being written to by what server last. On websvs1 I created a disk that almost filled the websvs storage up.. I then manually added that disk to websvs2 by editing the configs by hand. after restarting both websvs1&2 the same disk then appeared to both VM's on different physical servers. I then built the latest Linux kernel and OCFS2 drivers, then formatted and mounted the shared disk RW on both servers simultaneously and linked /var/www and /var/vmail to it. Both websvs1 & websvs2 then were setup for uCarp failover on both there internal and external interfaces.. I did a dual-primary on the external interface so each server could fail over to the others external IP. On the internal carp IP I ran a NFS server with /var/www. On the external carp IPs I am running HA-Proxy configured to monitor Web & Mail. HA Proxy weights the local server twice as high as the remote server if both are available. I then added 2 more servers in ProxMox, one on each physical server.. neither have external IP's as they just get port forwarded in and NAT out. I named these VM's admin & users. Admin runs the ISPConfig3 Web Panel/phpMyAdmin & RoundCube over https only. Users is a shell server that mounts /var/www over NFS off the websvs internal carp ip. I then installed ISPConfig3 on each server, starting off on admin and installing the webgui but nothing else. I configured websvs2 and users to be a mirror of websvs1, locked down ssh on websvs1&2 and using HA proxy forwarded SSH ports on external interfaces to users.. All the servers got a grsecurity hardened kernel and users got especially hardened. MySQL is Primary-Secondary configuration with HA-Proxy trying to send everything to websvs1 as long as its available... If websvs1 goes away then write access is denied, this is the only glitch in the nirvana but I can live with it. Both websvs1 & websvs2 are redundant and instantly replace the other in an outage, admin & users can be live migrated w/out interruption to the opposite server in a planed maintenance event or manually brought back up on the opposing server after an unexpected outage.. (I am planing on doing that automatically soon) If anyone has any questions, needs clarity or wants to see examples go ahead and post up.. the performance is quite excellent by the way, I can easily saturate 1Gbit using httperf and keep on going. Cheers, -Ryan
