I posted another thread about a similar problem ( https://www.howtoforge.com/communit...rking-completely-correctly.75191/#post-353989 ), but I'm not yet sure if the two issues are related, fundamentally. I noticed excessive disk space consumption on a particular server, and upon investigation, it was due to nginx logs not being compressed each day. This had always worked fine, and then on January 13, logs stopped being compressed. Now, every new daily access log is left uncompressed indefinitely. The directory output looks like this: Code: # ls -lah /var/www/domain.tld/log total 6.2G drwxr-xr-x 2 root root 4.0K Feb 5 00:00 . drwxr-xr-x 8 root root 4.0K Oct 1 18:05 .. -rw-r--r-- 1 root root 4.5M Dec 17 00:05 20161215-access.log.gz -rw-r--r-- 1 root root 4.5M Dec 18 00:04 20161216-access.log.gz -rw-r--r-- 1 root root 3.0M Dec 19 00:04 20161217-access.log.gz -rw-r--r-- 1 root root 2.3M Dec 20 00:04 20161218-access.log.gz -rw-r--r-- 1 root root 4.7M Dec 21 00:04 20161219-access.log.gz -rw-r--r-- 1 root root 4.6M Dec 22 00:04 20161220-access.log.gz -rw-r--r-- 1 root root 4.8M Dec 23 00:04 20161221-access.log.gz -rw-r--r-- 1 root root 5.0M Dec 24 00:04 20161222-access.log.gz -rw-r--r-- 1 root root 4.5M Dec 25 00:05 20161223-access.log.gz -rw-r--r-- 1 root root 3.7M Dec 26 00:05 20161224-access.log.gz -rw-r--r-- 1 root root 3.1M Dec 27 00:04 20161225-access.log.gz -rw-r--r-- 1 root root 4.5M Dec 28 00:04 20161226-access.log.gz -rw-r--r-- 1 root root 5.5M Dec 29 00:05 20161227-access.log.gz -rw-r--r-- 1 root root 5.5M Dec 30 00:05 20161228-access.log.gz -rw-r--r-- 1 root root 5.2M Dec 31 00:04 20161229-access.log.gz -rw-r--r-- 1 root root 4.7M Jan 1 00:04 20161230-access.log.gz -rw-r--r-- 1 root root 3.1M Jan 2 00:04 20161231-access.log.gz -rw-r--r-- 1 root root 2.6M Jan 3 00:04 20170101-access.log.gz -rw-r--r-- 1 root root 3.9M Jan 4 00:04 20170102-access.log.gz -rw-r--r-- 1 root root 5.0M Jan 5 00:06 20170103-access.log.gz -rw-r--r-- 1 root root 7.6M Jan 6 00:08 20170104-access.log.gz -rw-r--r-- 1 root root 21M Jan 7 00:06 20170105-access.log.gz -rw-r--r-- 1 root root 22M Jan 8 00:09 20170106-access.log.gz -rw-r--r-- 1 root root 21M Jan 9 00:07 20170107-access.log.gz -rw-r--r-- 1 root root 20M Jan 10 00:06 20170108-access.log.gz -rw-r--r-- 1 root root 24M Jan 11 00:08 20170109-access.log.gz -rw-r--r-- 1 root root 24M Jan 12 00:06 20170110-access.log.gz -rw-r--r-- 1 root root 24M Jan 13 00:05 20170111-access.log.gz -rw-r--r-- 1 www-data root 341M Jan 13 00:00 20170112-access.log -rw-r--r-- 1 www-data root 339M Jan 14 00:00 20170113-access.log -rw-r--r-- 1 www-data root 296M Jan 15 00:00 20170114-access.log -rw-r--r-- 1 www-data root 280M Jan 16 00:00 20170115-access.log -rw-r--r-- 1 www-data root 331M Jan 17 00:00 20170116-access.log -rw-r--r-- 1 www-data root 331M Jan 18 00:00 20170117-access.log -rw-r--r-- 1 www-data root 330M Jan 19 00:00 20170118-access.log -rw-r--r-- 1 www-data root 323M Jan 20 00:00 20170119-access.log -rw-r--r-- 1 www-data root 311M Jan 21 00:00 20170120-access.log -rw-r--r-- 1 www-data root 263M Jan 22 00:00 20170121-access.log -rw-r--r-- 1 www-data root 244M Jan 23 00:00 20170122-access.log -rw-r--r-- 1 www-data root 294M Jan 24 00:00 20170123-access.log -rw-r--r-- 1 www-data root 303M Jan 25 00:00 20170124-access.log -rw-r--r-- 1 www-data root 253M Jan 26 00:00 20170125-access.log -rw-r--r-- 1 www-data root 145M Jan 27 00:00 20170126-access.log -rw-r--r-- 1 www-data root 121M Jan 28 00:00 20170127-access.log -rw-r--r-- 1 www-data root 77M Jan 29 00:00 20170128-access.log -rw-r--r-- 1 www-data root 64M Jan 30 00:00 20170129-access.log -rw-r--r-- 1 www-data root 111M Jan 31 00:00 20170130-access.log -rw-r--r-- 1 www-data root 151M Feb 1 00:00 20170131-access.log -rw-r--r-- 1 www-data root 190M Feb 2 00:00 20170201-access.log -rw-r--r-- 1 www-data root 201M Feb 3 00:00 20170202-access.log -rw-r--r-- 1 www-data root 242M Feb 4 00:00 20170203-access.log -rw-r--r-- 1 www-data root 255M Feb 5 00:00 20170204-access.log -rw-r--r-- 1 www-data root 189M Feb 5 19:12 access.log -rw-r--r-- 1 www-data root 84M Feb 5 18:56 error.log -rw-r--r-- 1 root root 996K Jan 12 00:06 error.log.1.gz What could have caused this behavior? How shall I troubleshoot it? I have examined "/usr/local/ispconfig/server/scripts/create_daily_nginx_access_logs.sh", but there is no mention of compression there: Code: #!/bin/bash PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin FILES=/var/log/ispconfig/httpd/* for f in $FILES do mv $f/access.log $f/`date --date='yesterday' "+%Y%m%d"`-access.log &> /dev/null touch $f/access.log &> /dev/null done pkill -USR1 -u root nginx &> /dev/null What process is normally responsible for compressing old nginx access logs with ISPConfig?
Thanks for pointing me in the appropriate direction here, Till. I ran the cron script like this: Code: php -f /usr/local/ispconfig/server/cron_debug.php -- --cronjob=200-logfiles.inc.php It looks like this "worked", but only the log file from a couple days ago was gzipped (20170204-access.log.gz). Here' the new directory listing data: Code: # ls -lah /var/log/ispconfig/httpd/domain.tld/ total 6.1G drwxr-xr-x 2 root root 4.0K Feb 7 00:00 . drwxr-xr-x 4 root root 4.0K Oct 1 19:43 .. -rw-r--r-- 1 root root 2.6M Jan 3 00:04 20170101-access.log.gz -rw-r--r-- 1 root root 3.9M Jan 4 00:04 20170102-access.log.gz -rw-r--r-- 1 root root 5.0M Jan 5 00:06 20170103-access.log.gz -rw-r--r-- 1 root root 7.6M Jan 6 00:08 20170104-access.log.gz -rw-r--r-- 1 root root 21M Jan 7 00:06 20170105-access.log.gz -rw-r--r-- 1 root root 22M Jan 8 00:09 20170106-access.log.gz -rw-r--r-- 1 root root 20M Jan 10 00:06 20170108-access.log.gz -rw-r--r-- 1 root root 24M Jan 11 00:08 20170109-access.log.gz -rw-r--r-- 1 root root 24M Jan 12 00:06 20170110-access.log.gz -rw-r--r-- 1 root root 24M Jan 13 00:05 20170111-access.log.gz -rw-r--r-- 1 www-data root 341M Jan 13 00:00 20170112-access.log -rw-r--r-- 1 www-data root 339M Jan 14 00:00 20170113-access.log -rw-r--r-- 1 www-data root 296M Jan 15 00:00 20170114-access.log -rw-r--r-- 1 www-data root 280M Jan 16 00:00 20170115-access.log -rw-r--r-- 1 www-data root 331M Jan 17 00:00 20170116-access.log -rw-r--r-- 1 www-data root 331M Jan 18 00:00 20170117-access.log -rw-r--r-- 1 www-data root 330M Jan 19 00:00 20170118-access.log -rw-r--r-- 1 www-data root 323M Jan 20 00:00 20170119-access.log -rw-r--r-- 1 www-data root 311M Jan 21 00:00 20170120-access.log -rw-r--r-- 1 www-data root 263M Jan 22 00:00 20170121-access.log -rw-r--r-- 1 www-data root 244M Jan 23 00:00 20170122-access.log -rw-r--r-- 1 www-data root 294M Jan 24 00:00 20170123-access.log -rw-r--r-- 1 www-data root 303M Jan 25 00:00 20170124-access.log -rw-r--r-- 1 www-data root 253M Jan 26 00:00 20170125-access.log -rw-r--r-- 1 www-data root 145M Jan 27 00:00 20170126-access.log -rw-r--r-- 1 www-data root 121M Jan 28 00:00 20170127-access.log -rw-r--r-- 1 www-data root 77M Jan 29 00:00 20170128-access.log -rw-r--r-- 1 www-data root 64M Jan 30 00:00 20170129-access.log -rw-r--r-- 1 www-data root 111M Jan 31 00:00 20170130-access.log -rw-r--r-- 1 www-data root 151M Feb 1 00:00 20170131-access.log -rw-r--r-- 1 www-data root 190M Feb 2 00:00 20170201-access.log -rw-r--r-- 1 www-data root 201M Feb 3 00:00 20170202-access.log -rw-r--r-- 1 www-data root 242M Feb 4 00:00 20170203-access.log -rw-r--r-- 1 root root 17M Feb 6 23:59 20170204-access.log.gz -rw-r--r-- 1 www-data root 242M Feb 6 00:00 20170205-access.log -rw-r--r-- 1 www-data root 292M Feb 7 00:00 20170206-access.log -rw-r--r-- 1 www-data root 1.4M Feb 7 00:07 access.log -rw-r--r-- 1 www-data root 2.4K Feb 7 00:06 error.log -rw-r--r-- 1 root root 5.1M Feb 6 23:59 error.log.1.gz I "fixed" the existing situation with this: Code: find /var/log/ispconfig/httpd/domain.tld/ \( -name '*.log' \) -exec gzip --verbose {} \; That frees-up some 9GB of disk space on this server. I'd like to be sure that the log gzipping is functioning in a reliable way going forward. Should I simply check on it tomorrow and see if logs continue to be gzipped on schedule? And why might it have happened in the first place? This server has never been powered-off for any period of time, or subjected to any other event that would cause scheduled cron scripts not to run (at least to my knowledge). Thanks!
yes. I have no idea. If it still does not run, try this: edit the root crontab with: crontab -e insert a comment like "# test" and save, this should trigger a reload of the crontab. I've seen that cron for whatever reason lost a cronjob and triggering it like this made it being executed again.
Just so I know whether or not "touching" the crontab fixes the issue, I am going to wait until after midnight on this server (about two hours from now) before doing so. In the meantime, the file contains the following at the moment: Code: MAILTO = [email protected] * * * * * /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done * * * * * /usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 0 0 * * * /usr/local/ispconfig/server/scripts/create_daily_nginx_access_logs.sh &> /dev/null The contents look correct, I assume... we'll see what happens at midnight. If the job doesn't run, I'll touch the crontab and see what happens the following night.
Thanks, Till. Everything looks correct again today. Is it possible that once the log compression is "out of sync", it never recovers without manual intervention? I don't know what caused this condition in the first place, but it would be nice if successive cron runs would simply make the state consistent, no matter how out-of-sync it may have become. But that might asking a lot; to do so seems rather complicated!
