log files, rotation and emailing them

could you plz look it up? I just realized I have a localhost directory there as well. I would be very grateful to get rid of this problem

 

like suggested in another thread I used my local installation of php instead of the one that came with ispcfg and got another error than before:

the part with the array is gone now, but what about the remaining errors?

 

I had this localhost directory problem.
As a workaround, I edited line 122 in /root/ispconfig/scripts/shell/logs.php to

Code:

if(trim($virtual_host) != '' && trim($virtual_host)!='localhost') {

instead of

Code:

if(trim($virtual_host) != '') {

and my stats are now compiled daily.

 

thx, that worked for me, here is the result:

BUT I am missing the logs from 5 days which lay in between,... anyway thx, its ok so far, I hope it will continue to work automatically now

 

I'm honoured !

 

sorry to disturb again the peace of this threat but my stats stopped compiling again on the 25. of april and since then the logfiles have been growing again. on the 28. of april I have upgraded ispconfig to the latest version so that is not the reason as they stopped working before...

here is what a manual execution of webalizer gives me now:

also here is whats inside my /var/www:

Please help me, this can't go on like this.

 

http://www.howtoforge.com/forums/showthread.php?t=562&page=7

 

one more question,

I see that everything is fine now, statistics compiling, no error reports but still the logfiles of my sites are aroung 1.1GB although I set max log size to 500 which means 500MB if I am not wrong.

any hints? how to test or enforce this? I guess if they keep growing I'll get problems soon enough.

 

I'm sorry I overlooked this till now.

Yes, I made a directory /home/notwww/web0/ and made a softlink called "localhost" to there in /home/www/

So the script splits out stuff destined for localhost and puts it in /home/notwww/web0/log/2006/05/

What appears in there is just the evidence of hacking attempts. This is a recent sample:

64.246.188.65 - - [15/May/2006:11:45:55 +0100] "GET / HTTP/1.0" 200 895 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)"
64.246.188.65 - - [15/May/2006:11:45:55 +0100] "GET / HTTP/1.0" 200 895 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)"
66.221.207.46 - - [15/May/2006:14:48:01 +0100] "GET /a1b2c3d4e5f6g7h8i9/nonexistentfile.php HTTP/1.0" 404 1098 "-" "-"
66.221.207.46 - - [15/May/2006:14:48:02 +0100] "GET /adxmlrpc.php HTTP/1.0" 404 1098 "-" "-"
66.221.207.46 - - [15/May/2006:14:48:02 +0100] "GET /adserver/adxmlrpc.php HTTP/1.0" 404 1098 "-" "-"
66.221.207.46 - - [15/May/2006:14:48:02 +0100] "GET /phpAdsNew/adxmlrpc.php HTTP/1.0" 404 1098 "-" "-"
66.221.207.46 - - [15/May/2006:14:48:03 +0100] "GET /phpadsnew/adxmlrpc.php HTTP/1.0" 404 1098 "-" "-"
66.221.207.46 - - [15/May/2006:14:48:03 +0100] "GET /phpads/adxmlrpc.php HTTP/1.0" 404 1098 "-" "-"
66.221.207.46 - - [15/May/2006:14:48:03 +0100] "GET /Ads/adxmlrpc.php HTTP/1.0" 404 1098 "-" "-"
66.221.207.46 - - [15/May/2006:14:48:03 +0100] "GET /ads/adxmlrpc.php HTTP/1.0" 404 1098 "-" "-"
66.221.207.46 - - [15/May/2006:14:48:04 +0100] "GET /xmlrpc.php HTTP/1.0" 404 1098 "-" "-"
66.221.207.46 - - [15/May/2006:14:48:04 +0100] "GET /xmlrpc/xmlrpc.php HTTP/1.0" 404 1098 "-" "-"
66.221.207.46 - - [15/May/2006:14:48:04 +0100] "GET /xmlsrv/xmlrpc.php HTTP/1.0" 404 1098 "-" "-"
66.221.207.46 - - [15/May/2006:14:48:04 +0100] "GET /blog/xmlrpc.php HTTP/1.0" 404 1098 "-" "-"
66.221.207.46 - - [15/May/2006:14:48:05 +0100] "GET /drupal/xmlrpc.php HTTP/1.0" 404 1098 "-" "-"
66.221.207.46 - - [15/May/2006:14:48:05 +0100] "GET /community/xmlrpc.php HTTP/1.0" 404 1098 "-" "-"
66.221.207.46 - - [15/May/2006:14:48:05 +0100] "GET /blogs/xmlrpc.php HTTP/1.0" 404 1098 "-" "-"
66.221.207.46 - - [15/May/2006:14:48:05 +0100] "GET /blogs/xmlsrv/xmlrpc.php HTTP/1.0" 404 1098 "-" "-"
66.221.207.46 - - [15/May/2006:14:48:05 +0100] "GET /blog/xmlsrv/xmlrpc.php HTTP/1.0" 404 1098 "-" "-"
66.221.207.46 - - [15/May/2006:14:48:06 +0100] "GET /blogtest/xmlsrv/xmlrpc.php HTTP/1.0" 404 1098 "-" "-"
66.221.207.46 - - [15/May/2006:14:48:06 +0100] "GET /b2/xmlsrv/xmlrpc.php HTTP/1.0" 404 1098 "-" "-"
221.167.208.202 - - [15/May/2006:18:17:16 +0100] "GET / HTTP/1.0" 200 895 "-" "-"
83.104.39.2 - - [15/May/2006:20:54:21 +0100] "HEAD / HTTP/1.0" 200 - "-" "-"
58.68.4.26 - - [16/May/2006:03:29:12 +0100] "GET //README HTTP/1.1" 404 1100 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
58.68.4.26 - - [16/May/2006:03:29:13 +0100] "GET /horde//README HTTP/1.1" 404 1100 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
58.68.4.26 - - [16/May/2006:03:29:14 +0100] "GET /horde2//README HTTP/1.1" 404 1100 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
58.68.4.26 - - [16/May/2006:03:29:15 +0100] "GET /horde3//README HTTP/1.1" 404 1100 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
58.68.4.26 - - [16/May/2006:03:29:15 +0100] "GET /horde-3.0.9//README HTTP/1.1" 404 1100 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
58.68.4.26 - - [16/May/2006:03:29:16 +0100] "GET /Horde//README HTTP/1.1" 404 1100 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
62.233.144.195 - - [16/May/2006:14:21:46 +0100] "GET //README HTTP/1.1" 404 1100 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
62.233.144.195 - - [16/May/2006:14:21:47 +0100] "GET /horde//README HTTP/1.1" 404 1100 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
62.233.144.195 - - [16/May/2006:14:21:47 +0100] "GET /horde2//README HTTP/1.1" 404 1100 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
62.233.144.195 - - [16/May/2006:14:21:47 +0100] "GET /horde3//README HTTP/1.1" 404 1100 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
62.233.144.195 - - [16/May/2006:14:21:47 +0100] "GET /horde-3.0.9//README HTTP/1.1" 404 1100 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
62.233.144.195 - - [16/May/2006:14:21:47 +0100] "GET /Horde//README HTTP/1.1" 404 1100 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
83.104.39.2 - - [16/May/2006:15:58:35 +0100] "HEAD / HTTP/1.0" 200 - "-" "-"

I suspect that 2.2.2 has changed how localhost accesses are logged, but I've not upgraded yet.

 

still my logs are growing and ignoring the max logsize I entered...

is there a log processing file I can manually run to see if I get any errors? there was something called logs.php as far as I remember. right?

 

Yes. Run

Code:

crontab -l

to find the appropriate commands.

 

I originally installs ISPConfig version 2.2.3 and recently upgraded to 2.2.6

My Webalizer statistics are compiling nicely and I have no problem with those.

However, I have a user with a quota of 200MB and the logs are set to take up the default max. of 30%. In my opinion that should therefore be no more than 60MB

Looking at the statistics for that client I see that their logs now, after a good 2 months of operation, are at 67MB.

crontab -l shows me this:
# ISPConfig
30 00 * * * /root/ispconfig/php/php /root/ispconfig/scripts/shell/logs.php &> /dev/null
59 23 * * * /root/ispconfig/php/php /root/ispconfig/scripts/shell/ftp_logs.php &> /dev/null
59 23 * * * /root/ispconfig/php/php /root/ispconfig/scripts/shell/mail_logs.php &> /dev/null
59 23 * * * /root/ispconfig/php/php /root/ispconfig/scripts/shell/cleanup.php &> /dev/null
0 4 * * * /root/ispconfig/php/php /root/ispconfig/scripts/shell/webalizer.php &> /dev/null
0,30 * * * * /root/ispconfig/php/php /root/ispconfig/scripts/shell/check_services.php &> /dev/null
15 3,15 * * * /root/ispconfig/php/php /root/ispconfig/scripts/shell/quota_msg.php &> /dev/null
40 00 * * * /root/ispconfig/php/php /root/ispconfig/scripts/shell/traffic.php &> /dev/null
05 02 * * * /root/ispconfig/php/php /root/ispconfig/scripts/shell/backup.php &> /dev/null

And I have changed the defaults in php.ini to allow for more time (600 and 120secs respectively) and more (128MB) memory.

Still the logfiles continue to grow.

Any suggestions what might be the cause and, more importantly, what I can do about it? Like I said, Webalizer creates the statistics without a problem.

 

My suggestion is to try changing the 120 seconds to 240 seconds, but I'm not sure it will work. What I tend to do it to go round each month, deleting the previous month's log directories. It's not ideal, because some logs get past 500MB in a month.

As I type, I'm running the script with 600sec / 240sec / 96MB to see if that manages to trim the logs - there should be no new entries to parse becasue I didn't run logs.php first.

 

That took a long time to finish, but it produced some interesting results. Amongst the other entries, there were these two:

2069754 records (2069754 ignored) in 113.97 seconds, 18160/sec
2359521 records (2359501 ignored) in 184.96 seconds, 12756/sec

So setting the 240 seconds was not so high - it needed 185 seconds for one file - but the log stayed at 567MB, and the script is definitely not reducing the log size. Perhaps I need to increase the 600 sec to 6000 sec?

Once webalyzer has run, there's nothing to stop you deleting the log file, though.

 

It has set_time_limit(0); at the beginning already, but I'm not sure that works, because after changing the timeout in php.ini even higher, it has finally managed to trim the logs in last night's run - even though I reduced the memory allocation.

Should the line at the beginning of webalizer.php be time_limit(0) instead of set_time_limit(0): ?

 

It must be

Code:

set_time_limit(0);

 

Thanks. That is what is in the file, but wasn't allowing the scripts to finish. Increasing ther time limits seems to have sorted it out, but I suspect that the whole machine is running more slowly all the rest of the time.

After tonight's run I'll set the limits back and see if it speeds up again.