Somehow, the nodes in my cluster are responding differently. When I have no websites defined in ISPConfig, then it all works as expected. If I "curl [ip of node0]" or "curl [ip of node1]" then I get back the same response, which is the default vhost at /var/www/html. But when I create a website, then what happens is that "curl [ip of node0]" returns the newly created website (the ISPConfig "welcome!" page), while "curl [ip of node1]" returns the default vhost at /var/www/html. This is a problem because HAProxy is checking whether the nodes are up by querying for the file "healthcheck.html" at /var/www/html, as the idea is that the raw IP address of a node should always return the default website at /var/www/html. Node1 is doing what's expected, but Node0 is returning the newly created website instead. So, as Node0 is not returning the default website when called with its IP address, HAPorxy's not able to access "healthcheck.html" and considers the node has died, so removes it from the pool of real servers. In order to check that there's no ordering issue, I've modified the "apache2.conf" so that it manually includes the default vhost and ispconfig vhost files first, then it does "IncludeOptional sites-enabled", ensuring that it's just not possible for any other site to usurp the default position from default (this is probably a good idea anyway, lest some joker try to register the "00000-aaaa.com" domain name to be clever). There is a slight complication to this picture. The load balancing node running HAProxy is also running Apache as well, but this is for the ISPConfig interface itself and that's on port 8080 (and I've made sure that "ports.conf" only has "listen 8080" and no "listen 80" anywhere).
Has the vhost file of this site been created on all nodesby ispconfig? Check sites-available directory and the sites-enabled directory if it is actvee. Did you select * as IP address for the vhost, if not, then change it to *. Beside that, check if enabled the connect userid to web id option under system > server config > web for all nodes, this ensures that the website users on all nodes get the same linux uid.
Ah-ha. For some inexplicable reason, "enable * wildcard" was disabled on the one server and not the other, so when I created the website, as there was no "*" option on one of the servers, it was defaulting to the IP address of node0. Which I failed to notice had changed. To be honest, I might hack the ISPConfig interface and hide that field, forcing it to always be "*" or something. In our specific case, there would never be any reason for it to be anything other than that, and having that field there is just asking for trouble. I get that it could be useful in the generic situation, but all it's doing for me is causing problems. (And, yeah, I connected the userid to webid in the options already.) Thanks, Till. You're a star.
p.s. If I might quickly sneak in an extra question, where does ISPConfig store the "index.html" and error files that get copied into a newly minted website? So we can create our own branded variants of these with our name, logo and such added.
