[Solved] AWstats Issue

I'm getting an email sent from "www-data" that informs me that AWstats is having troubles.

The gist of this is that it's saying that "/var/log/ispconfig/httpd/somewebsite.tld/yesterday-access.log" doesn't exist or, in another case, that "/var/log/ispconfig/httpd/otherwebsite.tld/access.log" doesn't exist.

And, sure enough, when I browse to those directories, the "access.log" is missing and the problem with "yesterday-access.log" is that it's a symlink to an YYYYMMDD-access.log but the problem is that the log it's trying to point to is "YYYYMMDD-access.log.gz". The file is actually gzipped, by the symlink doesn't include the ".gz" extension.

A possible reason why this has happened is that ISPConfig makes a bunch of mounts in /etc/fstab to the access logs. The thing is that it uses "nobootwait" and this is no longer supported on Ubuntu 16.04. It uses "nofail" instead.

I did have to reboot the server in question and, as "nobootwait" is not valid in 16.04, this parameter is ignored and it does wait for these on boot. Thing is, all my website files are, in fact, mounted from NFS, so the waiting can effectively lock up the server, depending on whether NFS fails to come up first. I know you shouldn't, but I had to force it off by holding down the power button for 4 seconds.

To get around this, I was forced to boot up with the network cables unplugged - if the network can't be raised at all then there's no waiting around for countless timeouts - then I commented out the /etc/fstab lines and rebooted with the network restored. Then I looked up "nobootwait" and discovered that this no longer works on 16.04 and you should use "nofail" instead. So I made the changes and reinstated things.

But, yes, in doing so, it seems something somewhere has come unstuck and the log files are fubar. So AWstats is not seeing what it should and error-ing out periodically.

(I'll overlook the "nobootwait" bug, if you overlook my ungraceful way of getting the server to reboot - but, well, it's a server and I can't leave it down too long. It shouldn't go down at all. So you can hopefully understand the slightly panicked rush to get around this problem of it locking up on boot. I did give it a good 5 minute wait initially.)

Anyway, any ideas of how to fix this?

 

Two things.

First of all, where in the source code is the "nobootwait" added to "fstab"?

Because though I manually fixed the "fstab" to use "nofail" instead - and it does now boot up without hanging - the problem going forward is that any new websites that are created are created with "nobootwait" and I don't want to be manually fixing the bind mounts each and every time.

So if I know where in the source code that's added, then I can hack it to add "nofail" instead.

Secondly, I'm just wondering if there's a simple way to restore the bind mounts. Because they've obviously gotten screwed up in my slight panic to quickly fix the hang on boot. Can they be restored by, for example, disabling the stats in the website settings and then re-enabling them again? Would that work?

 

Actually, strike the first question. I used "grep". It's in the apache and nginx plugins.

 

I've missed the obvious.

I'm running a multi-server setup with shared storage. But what is being shared for the websites is "/var/www" (and "/var/vmail" for the emails).

What isn't being shared is "/var/log/ispconfig". So each node is seeing its own local "/var/log/ispconfig" and this means they all have their own separate access and error logs.

But the storage server is only a web server for technical reasons - e.g. mirroring - and does not actually deal with outside requests. So it does not have an "access.log" at all. And this is what AWstats is falling over with, as it tries to process things on the storage server. It looks for "access.log" and it simply isn't there.

The real underlying problem is here is that "/var/log/ispconfig" isn't being shared. In setting up the shared storage, I confess that I just plain forgot about the logs. In the case of the access and error logs - as found in "/var/log/ispconfig/httpd" - then that should be really be shared too, as I don't want per-server stats but website stats. Even if this wasn't causing problems, then I'd want it to be shared anyway.

Would it be safe to simply mount "/var/log/ispconfig/httpd" from the shared storage to the other nodes?

 

Well, I've tried it and the moment of truth is 15:50, as that's when the AWstats cron job should run next.

Although it should be good because the error messages are about not finding "access.log" and "yesterday-access.log" but I've checked and they all now exist from every web server's perspective.

 

Okay, it almost works.

I have one remaining issue, which is that it's saying that "SiteDomain" is not defined in "awstats.conf".

But this is the main configuration file. It gets included by the other files, which do define their own respective "SiteDomain".

This suggests that the cron job is trying to process the awstats.conf file itself?!?

 

Yes, I'm beginning to strongly suspect that the server hasn't been set up properly.

 

Yeah, I remember doing that when I was setting up the servers. But I guess it's possible - because, with a cluster, you're repeating the same "perfect server" procedure multiple times - that I accidentally skipped that step on this one machine or something. I'll check when I get into work later today.

 

Yes, that's what it was.

I must have somehow missed doing it on that particular server. The other servers in the cluster have it correctly commented out.

Oh well, that oversight did draw my attention to the fact that the access and error logs were per-server and I need them per-website, so they ought to be shared over the network.