Hi guys, early last week the master server (in an ISPConfig multiserver setup) died, and unfortunately all the latest system images (5 total!) at my vpshost were faulty (0kb size). First option was to try and swiftly rebuild everything on a new, clean vps and I contacted Florian for this (he didn't have much time and it was a weekend), but this resulted in a very unstable environment in which mysql kept crashing. Since I haven't been able to reach him since then, we were forced to use a system image that was 2 weeks old (luckily our vps host found this) and go from there. Environment is a multiserver setup (ISPConfig v3.1) with a master server (websites, owncloud, roundcube, phpmyadmin), slave mail server, slave server for particular client (websites, mail): master server: debian jessie, mysql 5.5.58, php 5.6.30, apache 2.4.10 slave main mailserver: debian jessie, mysql 5.5.59, php 5.6.33, postfix/dovecot slave client server: debian wheezy, mysql 5.5.59, php 5.4.45, apache 2.2.22, postfix/dovecot Since the image restore all the websites (databases/website folders) on the master (web)server have been updated with offsite backups. However over the course of those two missing weeks (two week old system image) several things were of course added/removed/changed from within the ISPConfig backend. For example: several mailboxes/domains/clients/websites were created and are still currently active on the mailserver, several postfix/content whilelist filters were added, etc. My main question therefore is: 1. should I simply enable the ISPConfig crontab jobs again (I was advised by Florian to uncomment these until the master server was working well again) on the slave servers and 2. then run the resync tool (Maildomains, Mailboxe, Mailfilter, Client and reseller from both the slave mail server and slave client server) from the ISPConfig master backend? As far as I understand this should preserve the existing 'new' mailboxes on the slave mail server? I have not manually recreated the clients/accounts associated with these 'new' mailboxes yet, since I assume they will be recreated when using the resync tool. Is this assumption correct? And if yes, what happens with the website folders/database that used to exist for these users (I assume I have to recreate those)? Will any of the manual edits (see below) on the current servers interfere with/be problematic for the resync process? If any of the below edits/changes will interfere with the Resync process or if any of them will be overwritten or not through the Resync process, I'd love to hear it. I am a total Resync newbie, as I'm sure is very obvious For the last few years everything has been smooth sailing with ISPConfig and our vps hoster until this happened, and now I'm very conscious about my non-skill level with ISPConfig when something out-of-the-ordinary happens. I also wouldn't mind paying for remote help / check-up, but since I haven't gotten a reply from schaal-it since last week's episode I'm not sure where to turn for this. Current master server Manual changes made since the old image restore on current master server: ISPC Set folder permissions on update: ON (regarding the www-data website folder changes, see point 2. below under Odd settings/behaviour) ISPC Default php handler: php-fpm (used to be fast-cgi) ISPC Backup mode: by webuser as zip (used to have the root option selected for some reason) >>> switched this back to root option since with ISPConfig v3.1 a lot of website folders/files are not owned by webuser group ISPC Makes SPDY available: checked (turned on) FastCGI config syntax is set to Old, shouldn't this be New (apache 2.2)? Changed the website backup time Changed pm.max_children to 15 and memory_limit = 256M in custom php.ini for one of the bigger sites, which was switched to php-fpm from fast-cfgi and (re-)added a database-(user) for this site 40+ new roundcube mailfilters by clients waiting to be propagated to the mailserver I also temporarily disabled automysqlbackup cron and munin(-node) in cron.daily / cron.weekly and temporarily disabled quota cron in cron.daily (since /var/backups from the old master server are currently still placed in /root/OLD/ and would count towards user quota's). Slave main mail server Edits since image restore on master server: Temporarily disabled ispconfig crontab jobs Manual edit: set slave mailserver to inet_protocols = ipv4 (someone was having problems with a particular mailserver only accepting ipv4 connections) Manual edit: deleted an entry from /etc/postfix/body_checks Manual edit: deleted reject_rbl_client dnsbl.sorbs.net from main.cf (multiple gmail servers were listed by sorbs today, never seen that before!) ... I should probably also delete sorbs from system > server config > mail > RBL Manual edit: removed from reject_unknown_client_hostname from main.cf Edits before crash: Several whitelist entries were added before the master server crash (both postfix and content filters) via ISPConfig that are no longer visible in ISPConfig master backend Several mailboxes/domains created via ISPConfig were created before the master server crash/after the used restore image, which still exist/are active on the mail server but do not exist in the master server ISCPonfig master backend Slave client server Edits since crash/image restore: temporarily disabled ispconfig job crons Manual edit: deleted reject_rbl_client dnsbl.sorbs.net from main.cf (multiple gmail servers were listed by sorbs today, never seen that before!) ... I should probably also delete sorbs from system > server config > mail > RBL Manual edit: removed from reject_unknown_client_hostname from main.cf Edits before crash/image restore: Several whitelist entries were added before the master server crash (both postfix and content filters) via ISPConfig that are no longer visible in ISPConfig master backend If any of the above edits/changes will interfere with the Resync process or if any of them will be overwritten or not through the Resync process, I'd love to hear it. I am a total Resync newbie, as I'm sure is very obvious For the last few years everything has been smooth sailing with ISPConfig and our vps hoster until this happened, and now I'm very conscious about my non-skill level with ISPConfig when something out-of-the-ordinary happens. I also wouldn't mind paying for remote help / check-up, but since I haven't gotten a reply from schaal-it since last week's episode I'm not sure where to turn for this. Odd settings/behaviour I also noticed some peculiar things when restoring the websites: Website permissions: for several websites the /tmp folders are set to 777, but most are set to 770 (which makes more sense to me)? Updating wordpress plugins etc now changes group to www-data instead of client? Is this an ISPC v3.1 exception, which perhaps also has to do with changing sites to php-fpm? (security level has always been set to High AFAIK) One particular /web folder is set to 755 instead of 711 (no idea how this happened) Two websites for which the database is not being backed up, but the webfolders are being backed up? (joomla db's, not large) To do (note to self) Things to do as soon as the current ISPC multiserver setup is resynced/working normally again: Apt-get upgrade on master server Incorporate https://schaal-it.com/current-rules-spamassassin-schaal/ and https://blog.schaal-24.de/mail/postscreen-im-kampf-gegen-spam/?lang=en Add domains and ip-adresses for gmail/google and .junkemailfilter.com (backupmx) to postfix whitelist in ISPConfig Upgrade Mysql from 5.5.x to 5.6.x (anything to watch out for?) >>> WPML WP-plugin requires it Upgrade to ISPC 7.11 Add PHP 7.0 next to 5.6 (for websites and ISPC itself) Install Let's Encrypt Sorry for the long rant! Best, Ronald
Resync is doing basically the same that would happen when you edit the settings of a site, change a value and then press save. So a safer method than using resync might be to start with a less important site, open the settings of that site, e.g. increase quota by 1 and press save. Then wait until the change has propagated and check if the site is working fine.
Hi Till, ty for the reply. I assume what you wrote is only applicable for the websites. Almost all of the websites are running on the affected main server and these have all been put back into place from (newer) offsite backups. There are a few websites on the client slave server, but these haven't been affected (as of yet) by the crash and also haven't changed since then. Quota crons on the master server are disabled at the moment, as stated, so quota's do not have my highest priority right now... I just figured that once everything else is in working order again I would simply remove the whole backup in root/OLD/ and switch the quota crons back on after. Unless of course I'm on the wrong track with this line of thought. I'm probably mostly worried about what re-enabling the ispconfig cronjobs on the slave servers (even without using the Resync Tool) will do to the operation of the slave mailserver (since a few mailboxes are up and running there but currently missing in the master ISPConfig). I guess my first question should have been if the ispconfig cronjobs work/sync two-ways or just from master > slaves? I assume not having the aforementioned existing (slave) mailboxes in the main server ISPConfig backend won't get them deleted when turning on the cronjobs, but I'm not confident enough to try it yet I also assumed that the settings for websites running on the main server are not synced to the slave servers by the ispconfig cronjobs on the slave servers, but also this I'm not sure about. Still feeling very out of my depth since it's all production servers and I can't just go for trial-and-error, perhaps I should try Florian again?
Just from master to slave. What you might do is that you set the value of the 'updated' column of all servers to the max. value of the datalog_id column from sys_datalog table. What the slaves are doing is: they ask the master to receive all transactions from sys_datalog where datalog_id < the value in the updated column of their own server record on the master. So by setting updates to max datalog_id, no server should start pulling any old transactions from master when you enable the server.sh cronjob.
Thank you for that clarification, Till. But, considering my nervouseness on this subject, I think I'll try and contact Florian again and try and get some hands-on help. If you have anyone else you can recommend, please do. That only gives me more options to get this fixed asap
Florian is a well-versed system administrator and ISPConfig developer, so I'm sure he will be able to help you with that.
Hi Till, Sorry to say I haven't been able to get Florian to help out yet (busy / not easy to get a hold of). So I have to ask again if you perhaps know of someone else that could help out with this, I'm stuck and by now need to get this figured out asap. Any help / suggestions are more than welcome! Thank you.
