Configuration - Ubuntu 16.04 server - DRBD 8.4.5 - Ispconfig 3.1.9 - 2 servers in cluster When I try to restart master server it hangs during a process (found in syslog) block drbd2: State change failed: Device is held open by someone Exactly the same behavior when I manually try to transfer resources to secondary server. Since this volume contains only websites (mounts to /var/www), I started to suspect apache. With a help of "lsof" I found out, that a process apache2 keeps open a few hundred files "error.log", which obviously prevent drbd from unmounting a volume. apache2 18473 www-data 50w REG 147,2 1477810 22151702 /var/www/clients/client25/web20/log/error.log Approx half of these files are zero size. Question - am I looking in the right direction at all ? - if yes, how to force apache not to keep these files open at all times ?
Normally, apache writes the error.log files to /var/log/ispconfig/httpd/<tmpl_var name='domain'>/error.log and /var/log/ispconfig/httpd/<tmpl_var name='domain'>/ is then mounted to /var/www/.... with a bind mount. Did you alter the vhost template so that apache writes to /var/www directly?
Templates are managed by Ispconfig, we didn't manually override any setting. The strange thing is as follows : every instance of apache service keeps open error.log for all sites. Example for one vhost only vlogger 6246 root 15w REG 147,2 77655 22157285 /var/www/clients/ client1/web23/log/20180220-access.log apache2 2754 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 3127 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 3129 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 3702 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 3718 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 3731 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 3733 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 3735 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 3736 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 4202 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 6815 root 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 6987 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 6988 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 6989 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 7344 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 7946 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 8308 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 10039 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 10040 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 10041 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 10054 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 10055 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 10056 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 10057 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 22756 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 22757 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 23643 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 25525 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 27986 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 29736 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log apache2 29742 www-data 40w REG 147,2 867006 22151780 /var/www/clients/ client1/web23/log/error.log I have no idea if this is normal or not.
That's normal. I just wonder why they appear under /var/www/... and not /var/log/.... When you take a look at one of the vhost files, does the error.log path is /var/www/... or /var/log/.... there?
Templates are managed by Ispconfig, we didn't manually override any setting. Where can I check these settings?
DocumentRoot /var/www/domain.tld/web ErrorLog /var/www/domain.tld/log/error.log Alias /error/ "/var/www/domain.tld/web/error/" where can I change this?
You are probably using a custom file already as the file that ships with ISPconfig 3.1.11 has the line: ErrorLog /var/log/ispconfig/httpd/<tmpl_var name='domain'>/error.log inside. Custom config file templates are in the folder /usr/local/ispconfig/server/conf-custom/
Unfortunately these folders are "empty" (error, index, install, mail). Only a file "empty.dir" in each folder. Interesting thing is that error logs are at expected location /var/log/ispconfig/httpd/<domain>/error.log only at approximately half of websites, others are found in /var/www/<domain>/log/error.log Search goes on...
I believe I found a reason... It seems that from some point on logs are directed to correct folder. In "/etc/apache2/sites-enabled/<domain>.vhost" file is entry ErrorLog. For some sites correct, for most of them not. Last year we migrated web server to new platform with fresh install of everything. It seems to me that vhosts added later are OK. I wish there is a way to correct wrong entries using ispconfig and not manual editing of vhost file...
Unfortuntely there's no end of strange situations... in most of the sites everything is OK, but there's a problem with some of them. There are "duplicate" log files in two locations (please see appacteh picture) Of course if I delete a file on either side, it disappears on both. And I can't find any symbolic link. Can somebody explain what is happening ?
That's exactöy as it should be. The log dir in /var/log/ispconfig/httpd/domain.tld is mounted to /var/www/.... with a bind mount to allow read-only access by the web user to his log files.
Thanks for clearing this out. It seems that original sin was attempt to store logs in /var/www/_website_/log folders (to sync everything using drbd) and not the location originally intended by ispconfig. But there's always "but"... which script / process is doing bind mount ? Because for most of the sites now logs exist only in /var/log/ispconfig/httpd/_website_/, but no change is reflected in /var/www/_website_/log Or to ask in a different way : what I have to do to normalize/initialize situation ? Regards
Logs get stored by ISPConfig to /var/log/ispconfig/httpd/_website_/ and not /var/www/_website_/log as logs shall not be stored in the web folder. The folder /var/www/_website_/log is just mounted with a BIND mount in /etc/fstab and this bind mount is added by the apache or nginx plugin at the time the website gest created. It works like this since ISPConfig 3 exists.
Things are getting clearer now. In /etc/fstab I've found entries for bind mounting log folders. But the problem is DRBD... in a moment when entries in fstab should be mounted, the source doesn't exist yet. DRBD mounts partition later on. Thanks for clearing things out. Will solve it with some script - at least temporarily, until I find permanent solution.
