Can I replicate ISPConfig configuration for simple recovery of the cluster?

This tutorial https://www.howtoforge.com/tutorial...abase-cluster-on-debian-8.4-with-ispconfig-3/
Seems to achieve complete replication except the ISPConfig panel. Which means, per my understanding, that after one of the two servers is destroyed/lost, new server with ISPConfig needs to be manually setup and then the replication needs to be set to be from remaining server to new server?

Can you please explain steps needed to do to recover above linked 2 server cluster, which lost one of its servers? Without having backup (only having one of the cluster members). Can you please explain the steps using names that I can find inside ISPConfig and reproduce it?

 

I thought that the cluster is made the way, that it is kind of self-healing were it is simple to replace any of its members using built in functions/script. I assume that the above linked cluster setup is useful to minimize the downtime until the master server is manually restored from backup, by switching to a replica/slave node IP in a DNS records of the domain name. Though it is then complicated for me to understand how is then synced replica/slave to master (since replica/slave SQL database got modified by website users). Any link to explanation about steps to do when master node is lost and to sync. the cluster again?

If admin needs to create custom master node backup solution (I understood that the replica/s does not need to be backup since it is synced from master), which master directories I should backup or which ones to skip, so i can restore the backup under the newly installed system, to which I will be restoring (my server provider does not provide visual KVM console for live ISO boot nor snapshots of the system)?

If you have any master node backup solution/script/commands, please share it.

 

And what happens if you lose all the nodes in the cluster? (rare, but does happen)

you need multiple backups in multiple locations (both network and geographic), and even then you don't have any security unless you've got a fully documented, and tested restoration process, and even then it's not 100%, massive geological events, nuclear war, overwhelming dns attacks, crowdstrike , etc could still take out access to all your backups. although in most of those cases i'd guess most of us would have more pressing issues than getting a broken webserver operational again.

 

and if I have backup, how to proceed then?

my project is not of that importance to handle such a case where each server is with different provider and location and all lost at same time, but thank you for your input!

I was hoping to have a geographically wide-spread cluster (across multiple countries), which may not always have latency under 200ms.