I'm still going through the installation guides and playing around with multi-server setups. I've noticed the configuration all seems to be stored in SQL databases and eventual consistency is... pushed via cron jobs (is that correct, maybe inotify)? There also seems some delay when applying changes (red dot status) sometimes taking a minute or so. I was just wondering if it has been considered to make use of better configuration -specific distributed stores like etcd and zookeeper? They are quite performant, designed as highly consistent, provide change notification events, strongly serialized, implement quorum checks for consensus, and have a much smaller footprint etc. How doe ISPConfig handle things like consistency with the SQL replication, operation order, high availability, etc? Is there more than just simple, small sized, configuration information in the SQL database (that warrants not using etcd or zookeeper as they are limited by item size)?
They are not better for this purpose. They would just duplicate the same information in a second place, while not being needed. Please see the various multiserver tutorials if you like to learn more about large-scale setups. ISPConfig is open-source software, so you can check out the source code if you want to learn how things work in detail.
Not better in what way? What advantages does a cron sync'd SQL database have over a central, API driven, small footprint, serialized, highly available "distributed consensus" solution based upon RAFT protocol? Overhead for etcd/zookeeper should be much smaller. Footprint should be much smaller (no need for multiple SQl servers, one on each system). Maintenance and upgrades would be simplified (no need to upgrade multiple individual SQL servers). Updates would likely be faster (though untested) as smaller bytes would be sent for changes, and changes implements upon even notification. Better high availability with automatic failover to master/leader (vs a single master SQL with manual promotion of SQL master).
You miss the important point that not just ISPConfig uses MySQL or MariaDB, but many other services in the stack use it directly for scalability too. E.g. the whole mail system. And we not only move the system config information around with MariaDB. Neither etcd nor zookeeper replaces a SQL server, so all information would have to be kept in two separate systems. That's all available for MariaDB, too, if you want to have it. But typically, most do not set it up like this, as no slave node depends on the availability of the master database, as ISPConfig replicates the relevant data automatically. So in the worst case, if you had a failure of the master, then only the GUI would be unavailable temporarily until you restore it, and in today's world, where one would use virtualisation anyway, such a full restore would just take minutes. But you can easily prevent the unavailability of the master GUI if you want by using HA for the MariaDB of the master and then connecting as many redundant GUIs to it as you want.
