Apache log rotation not configured in ISPConfig 3?

I've just realized that my webserver logs were piling up in the tens of gigabytes. I have checked the logrotate.d config for apache, but it is not configured for the log files in /var/log/ispconfig/httpd/, it only handles the default apache2 log directory, which is empty:

Code:

root@ispconfig:/etc/logrotate.d# cat apache2
/var/log/apache2/*.log {
        weekly
        missingok
        rotate 52
        compress
        delaycompress
        notifempty
        create 640 root adm
        sharedscripts
        postrotate
                if [ -f "`. /etc/apache2/envvars ; echo ${APACHE_PID_FILE:-/var/run/apache2.pid}`" ]; then
                        /etc/init.d/apache2 reload > /dev/null
                fi
        endscript
}

Furthermore, there is no trace of logrotate in the cron folders or crontab.

Is ISPConfig able to handle log rotation, or should I configure this myself, separately?

Logrotate.d seems to pose a problem, as I would have to add every site to it manually, as ISPConfig keeps the log files in separate folders.

 

Till, how is it possible that I had just over 5 months of logs, none of it deleted or compressed? We are running ISPConfig 3.0.1 currently.

The script you mentioned is not in crontab nor is it visible in /etc/cron.daily. Should I copy/link it over?

 

Second try

I have updated my ISPConfig to the latest version, but there is still no log rotation script (or anything connected to ISPConfig whatsoever) in the cron.daily folder or in /etc/crontab.

Here is the install script output:

Code:

Operating System: Debian Lenny/Sid or compatible

This application will update ISPConfig 3 on your server.
MySQL root password []: xxxxxxxxxxxxxxxx

Reconfigure Services? (yes,no) [yes]: no

Updating ISPConfig
ISPConfig Port [8080]:

chown: invalid user: `getmail'
Reconfigure Crontab? (yes,no) [yes]: yes

Updating Crontab
Update finished.

Do I need to reconfigure all services to enable the log rotation?

 

Ok, but how does it run if not from cron? How can I set the time it runs, so it would not interfere with my other nightly tasks on the server?

By the way nothing happened last night. Log files are still there, none is compressed or deleted.

 

still no luck

Well, I let it run for 2 days but nothing happened. The cron job is there, the script runs according to syslog, but it just doesn't compress the day old file and doesn't delete the month old one. Checked at several different sites, with log folders both empty and full with hundreds of files.

/var/log/ispconfig/cron.log shows nothing relevant.

What may be wrong? Any ideas how to nail the problem down?

 

why error.log don't rotate like access.log?

 

Till, any idea why on my machine the log rotation would not work?
Maybe a thing or two to check?

 

I have installed the file you provided, but still nothing: logs don't get compressed or deleted. Any idea what else to check?

 

Looks like I'm bumping an old topic, but forgive me.

I was curious about log rotating too after I saw error logs aren't rotated, but looks like future versions will have that too.

But I am concerned that cron jobs are in fact php files and thus depend on lenient PHP security settings. On my server for example, I have disabled exec function among others which is used in cron_daily.php.

Now of course following functions do not work.

Isn't this a rather severe oversight in ISPConfig 3?

 

Hello Till,

I watched into the given file but it seems that it is not exactly it that handles the log rotation.

In this file we have the compression of the file once it's already been renamed by another something and the generation of the statistics etc. etc. ...

So, if you could tell me what is the script / program which is, every day, re-generating the symlink from access.log -> yesterday.log to access.log -> today.log, it would be really neat, because I have a misfunctionning of the log rotation for one website and I would like to see what's hapening and why it is not happening for a particular website.


Thanks in advance.

 

Hello guys,

I've been thinking about all this and finally came with this conclusion : Get rid of Vlogger for Nginx.

Well, after all, nginx is significantly faster than Apache and it is known to handle correctly disk access.

But remains the problem of the filenames for Webalizer and AWStats, so I made that :

Create the directory /var/log/ispconfig/nginx and make nginx log into domaine.tld/access.log as Apache do for the httpd/ folder :

(configuration Nginx for a given server domain.tld) :

Code:

access_log  /var/log/ispconfig/nginx/domain.tld/access.log;

In /etc/logrotate.d/nginx, add at the end of file :

Code:

/var/log/ispconfig/nginx/*/access.log {
        daily
        missingok
        nocompress
        rotate 1
        notifempty
        sharedscripts
        postrotate
                [ ! -f /var/run/nginx.pid ] || kill -USR1 `cat /var/run/nginx.pid`
                /usr/bin/nohup /usr/local/ispconfig/server/logrotate_nginx.sh >> /var/log/ispconfig/cron.log
        endscript
}

Now create the two following files :
/usr/local/ispconfig/server/logrotate_nginx.sh

Code:

#!/bin/sh

PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin

/usr/bin/php -q /usr/local/ispconfig/server/logrotate_nginx.php

/usr/local/ispconfig/server/logrotate_nginx.php

Code:

<?php

require('lib/config.inc.php');
require('lib/app.inc.php');

set_time_limit(0);
ini_set('error_reporting',E_ALL & ~E_NOTICE);

$yesterday=date('Ymd', time()-86399);

chdir($conf['ispconfig_log_dir']);

$dirs = glob('nginx/*/');
foreach($dirs as $dir) {
        // Put the content of the access.log of today in the file supposed to be used for yesterday :
        file_put_contents($dir.$yesterday.'-access.log', file_get_contents($dir.'access.log.1'));
        // Empty the current access.log to have only statistics about today at next daily logrotation :
        unlink($dir.'access.log.1');
}

?>

Now remains two choices :

Either you modify server/cron_daily.php in order to go through nginx directories too.
Either you modify my script and replace "$dir.$yesterday.'.log'" by something like "httpd/".str_replace('nginx/', '', $dir).$yesterday.'-access.log' and this will copy the file into the httpd directory which is parsed by the cron_daily.php

 

I know that bumping a thread that is almost a decade old is really bad, but I was debugging one of my websites and suddenly noticed that the error.log was over 300 MB (!!!) in size, and extended back four years or so. Clearly there was something wrong!

/usr/local/ispconfig/server/cron_daily.php doesn't exist any longer, but we have /usr/local/ispconfig/server/cron.php which will read everything in /usr/local/ispconfig/server/lib/classes/cron.d, including a file called 200-logfiles.inc.php. Now this isn't something to 'configure', but rather directly hacking into the code, which I prefer to avoid, lest it'll be overwritten on future versions of ISPConfig 3.x

Interestingly enough, though, crontab -l shows the following line:

Code:

0 0 * * * /usr/local/ispconfig/server/scripts/create_daily_nginx_access_logs.sh &> /dev/null

which I believe I might have put there while following some instructions elsewhere... I can't really remember!
In any case, this script has the following code:

Code:

#!/bin/bash
PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin
FILES=/var/log/ispconfig/httpd/*
for f in $FILES
do
  mv $f/access.log $f/`date --date='yesterday' "+%Y%m%d"`-access.log &> /dev/null
  touch $f/access.log &> /dev/null
done
pkill -USR1 -u root nginx

so what I did was to add

Code:

  mv $f/error.log $f/`date --date='yesterday' "+%Y%m%d"`-error.log &> /dev/null
  touch $f/error.log &> /dev/null

before done, and that certainly works as expected.
I have no idea if this is the 'recommended' way of rotating the error logs, but at least it works for now...

 

Piping in (almost) two years later... the problem still persists, but I'm really confused — it's quite obvious that the ISPConfig cron jobs are being run, but why don't they rotate the logs (error.log and access-log)? Might this be related to webalizer/awstats 'reserving' the logs for their own statistics... and thus cron.php may skip all websites that have statistics turned on? I'm just wildly guessing, I'm not sure about anything any more...