Hi Till, hi forum, we are tasked by one of our customers to upgrade their Ubuntu 16.04 based ISPConfig setups. They have distributed their customers' web sites over several ISPConfig servers and while changing the underlying (virtual) hardware and upgrading the software, we also want to set up a more future proof server setup. The main tutorials we found are https://www.howtoforge.com/tutorial/ispconfig-multiserver-setup-debian-ubuntu/ and https://www.howtoforge.com/tutorial...abase-cluster-on-debian-8.4-with-ispconfig-3/ . The overall architecture we would like to have feedback on is the following: - we are using VMs on Azure in 2 separate regions (Germany West Central/Frankfurt and West Europe/Amsterdam) to increase our availability even in case of a region failure in Azure. - the VMs will be able to securely connect to each other using private IPs (for the Azure buffs in here: 2 VNets connected via peering) - the needed ports will be made available via the public internet, too - 1 panel server in Frankfurt - 2 web servers: 1 in Frankfurt, 1 in Amsterdam, set up as *mirrors* of each other. The main purpose for the VM in Amsterdam is to cover for the Frankfurt VM while this is undergoing an outage or maintenance while we still keep a tight SLA towards the end customers. - 2 mx and dns servers: 1 in FRA, 1 in AMS. We need them initially, as the customer has used them. We want to phase out own mx, dns servers and rather use dns and mx as a service (Azure DNS zones and AWS SES). - webmail will run on the FRA mx We also found a couple of helpful forum posts and understood: - the panel server hosting ISPConfig will not be replicated, as the web server will still server their content to the end users. One cannot make changes to ISPConfig while the panel is down (obviously) but using virtualization and backups that should be a rather minute problem. https://forum.howtoforge.com/thread...face-on-other-server-if-master-is-down.89151/ - NFS is needed to replicate the /var/www FS so that let's encrypt does not fail https://forum.howtoforge.com/threads/ubuntu-20-04-multiserver-setup.86872/ Open questions: - The web sites can be shops and other things having databases. We need a load balancer in front of our mirrored web servers to ensure that either -- active/active: each user only ever has a session on 1 web server or -- to use the web servers in an active/passive configuration, right? - should we integrate the panel on the 1st web server as we otherwise get a complex 3-party MySQL (MariaDB 10.6.9) multi-master setup? With 2 mysql master server we can simply follow the tutorial https://www.howtoforge.com/tutorial...ase-cluster-on-debian-8.4-with-ispconfig-3/2/ - how exactly would we make upgrades on the web server not being used in production? The links I found all recommend to upgrade the master server first, which would be the one serving the web requests in an active/passive scenario. - what do we need to do/know when we want to fail-over to the 2nd VM in an active/passive scenario? - general remarks and things we missed/should be very careful with? The core objective #1 is an easy transition from the current separate and outdated machines setup our customer has now. The core objective #2 is being able to restart one web server for updates at any time but still being "online" and serving our customers' customers web sites. Thank you!
Second thoughts about it: Can one really use mysql/mariadb in an active/active multi-master scenario where multiple copies of the same database on different servers can be written to? Only very few dbs do support this as the usual stance in relational dbs is rather conflict avoidance than (client side) conflict resolution. In case of an active/passive setup a load balancer would not be strictly necessary as one can control to which IPs/VMs the users are being redirected via DNS, or am I overlooking something here?
I would recommend active/active as you otherwise have one idle server all the time. And when the time comes and it shall become active, you'll see that Murphy's law applies and everything goes down I would recommend setting up the panel on a separate server with a regular MariaDB setup, so don't make the panel highly available. Your load balancer shall detect when it's down, but if you are doing maintenance you can always tell your load balancer not to use one of the 2 servers (depending on the exact software you use for load balancing). Set up the load balancer properly I see you want to move away from your own nameservers and email servers. If this is a hosting provider with zones for quite some clients, I'd strongly recommend to keep hosting your own servers, to 1) have everything in one panel and 2) be able to make easy batch changes through the database. But it really depends on the use case I guess. Yes, it is surely possible to host MariaDB as an active/active cluster. (Master/master). You are overlooking something here. When one of the server goes down and you have to manually change the DNS record, or even use a script to monitor it and change the DNS record for you, the DNS records will be cached. Even if you set up a TTL of 60 seconds there are DNS server that ignore that and cache it for longer. So your setup will still be down. Depending on how large these webshops/other web applications are and what the SLA agreement is, you might be better of not to use a master/master setup. If you are inexperienced with it it might lead to more downtime than just having a single server. Most of my servers are always up, unless I have to reboot for an upgrade, which I do at night and doesn't take more than 1-2 minutes.
Hey Th0m, I am working with the original problem poster. We tried creating a two server cluster with master-master MySQL replication and master-slave ISPConfig setup but faced a bug/minor problem while importing sites from a third server using ISPMigrate. The files import and site works fine on the main server, but on the slave server, the php-fcgi-starter (in /var/www/) files are not being copied "correctly". They are missing a path to "usr/lib/cgi_php7.0" on the slave server. IMO ISPConfig is responsible for replicating these files correctly on the slave but it fails to do so. Fixing these paths manually solves the problem, but shouldn't this be automatically done ?
Config files do not get copied between nodes, each ISPConfig instance creates them independently based on the settings that were made for that instance. If the fcgi files are wrong, then this is most likely related to wrong settings for the slave node or you modified config templates on the master and missed doing the same modifications on the slave. This is not related to the Migration Tool as all the Migration Tool does is to use the API to create things on the master, the slave node then connects to the master and fetch config transactions and reproduces them on the mirror slave node. If the PHP version that is causing the issue is an additional PHP version, then the problem might be that the additional PHP version is not present on the mirror (in the ISPConfig database). You can get a detailed insight what's going on on master and slave by using debug mode: https://www.faqforge.com/linux/debugging-ispconfig-3-server-actions-in-case-of-a-failure/
@till thanks for the reply. I found some relevant php settings in (accessing from master) Web Console > Settings > Server Config > Slave Server (fqdn) Fast CGI . Here I copied same settings as the Master, earlier they were blank. I also made sure other configs under Web are same. After this I resynced from Tools tab and rebooted the slave, but still the fcgi starter files are missing the lines to php cgi. Is there a way to force recreate those files after correcting the settings in the console? [EDIT] this is the error in error.log for apache on slave server : Code: .php-fcgi-starter: line 16: exec: -d: invalid option exec: usage: exec [-cl] [-a name] [command [arguments ...]] [redirection ...] and here is the difference between Master and Slave php_fcgi_starter file : Master : Code: exec /usr/bin/php-cgi7.4 -d open_basedir="/var/www/clients/client3/web3637/web:/var/www/clients/client3/web3637/private:/var/www/clients/client> -d upload_tmp_dir=/var/www/clients/client3/web3637/tmp -d session.save_path=/var/www/clients/client3/web3637/tmp -d sendmail_path="/usr/sbin/sendmail -t -i -f [email protected]" $1 Slave: Code: exec \ -d open_basedir="/var/www/clients/client3/web3637/web:/var/www/clients/client3/web3637/private:/var/www/clients/client3/web3637/tmp:/var> -d upload_tmp_dir=/var/www/clients/client3/web3637/tmp -d session.save_path=/var/www/clients/client3/web3637/tmp -d sendmail_path="/usr/sbin/sendmail -t -i -f [email protected]" $1
UPDATE : Apart from the missing path to cgi-binary executable as mentioned in my previous post, the SSL certificates are also not being copied over to the slave in some cases. After manually patching the php-fcgi-scripts we noticed some of the sites were working but others were still showing the same error Code: .php-fcgi-starter: line 16: exec: -d: invalid option . Digging into the .vhost files for these websites we found that they are missing the SSL directives (*443) and thus the patched php-fcgi-scripts were never accessed (accessing http version redirected to https, thus could not check otherwise). Since these vhosts files resides in /etc/apache2, they are not synced by unison either (which only syncs /var/www). Furthermore the SSL certificates are missing inside /etc/letsencrypt folder on slave. After syncing the vhost files and the ssl certificates we could finally access the websites on slave node. Interestingly, after just copying over the SSL certificates and "resync"-ing from ISPC console on master, the VHOST files are fixed automatically. To summarize, migrating sites into a cluster have following problems: 1. php-fcgi-starter files are missing path to php cgi binary files on Slave node, but are correct on master node. Since, as @till said, these files are created by the node's ISPC itself, we double checked that under System > Server Config > Slave > FastCGI and can confirm both the nodes have exact same settings. The slave is also set as a mirror of master as instructed in the guide. I found somewhere on the forum that after installing php we need to set the default to system default version using update alternative, I did that too but that didn't fixed it either. 2. The SSL certificates stored on master's /etc/letsencrypt are not copied over to slave due to which the vhost files on slaves are not properly configured leading to the same problem even after patching the php-fcgi-starter files. If it helps, the missing SSL problem is only present in websites which have their certificates stored in /etc/letsencrypt (but symlinked to /var/www/DOMAIN/ssl) and NOT for those sites which stores the SSL certificates in /var/www/DOMAIN/ssl directly and they work perfectly fine by just patching the php-fcgi-starter file (i.e their VHOST files are correct to begin with).
Continuing.. <A little information about how we setup the cluster> We primarily followed "https://www.howtoforge.com/installi...tabase-cluster-on-debian-6.0-with-ispconfig-3" guide but with some deviations. 1. We didn't installed Dovecot and XMPP. 2. We installed ISPC on master using the auto installer Code: wget -O - https ://get.ispconfig.org | sh -s -- --use-php=7.0 --use-certbot --use-ftp-ports=35000-50000 --lang=en --no-quota --unattended-upgrades and not as given in the tutorial, this is because it simplifies the installation. 3. We installed ISPC on slave using "https://www.howtoforge.com/tutorial...-8-4-jessie-apache-bind-dovecot-ispconfig-3-1" as asked in the tutorial but there were some steps which no longer work for ubuntu 20.04. These were steps for Let's Encrypt and Jailkit. We used steps from "https://www.howtoforge.com/tutorial...x-doveot-and-ispconfig/#-install-lets-encrypt" tutorial instead only for these two components. We also installed the latest ISPC (3.2) instead of 3.1 as asked in the tutorial. Note : We didn't use auto installed for slave since it disturbs the MySQL settings including root password which in turn disturbs the MySQL replication.
Why not follow the latest multiserver guide? https://www.howtoforge.com/tutorial/ispconfig-multiserver-setup-debian-ubuntu/ You state that you expect the LE certs to be copied over to the slave automatically, but that is not how it works. Setting up a highly available setup has been described several times here in the forum in the past months, I'd recommend reading that.
Understood, I think manually copying is the way to go then in this case. I started with it but found it bit more complicated and off topic than the other one, which was exactly what we needed. From reading I only find one difference that is the use of auto installer for all the nodes. I will give it a try and report back if the php-fcgi-starter issue is resolved.
No, it's not. Especially not when the second webserver could be used when the acme challenge is done. See the forum for several posts regarding such setups. I think it's logical that it's better to use a guide that states it supports a currently supported OS (Debian 10/11, Ubuntu 20.04) than a guide for Debian 6.
