I have them all. I tried to delete current and history files and run webalizer. No effect. I wrote a Python script to merge the files in chronological order and run mergelog | webalizer. It's telling me it cannot read March 1. Might be an old program reading a different format? Anybody here have experience restoring server logs? Please enlighten me.
Looks like /var/log/apache2 has a log for each week. These files will not work as far as I can tell. So I have two months of access logs generated by webalizer. I wrote a little Python script to run webalizer against each log file in my directory. The output follows… Code: Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140317-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 3146 records (3146 ignored) in 0.50 seconds Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140318-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 5817 records (5817 ignored) in 1.00 seconds, 5817/sec Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140319-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 5247 records (5247 ignored) in 1.00 seconds, 5247/sec Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140320-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 10326 records (10326 ignored) in 1.50 seconds, 6884/sec Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140321-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 7890 records (7890 ignored) in 1.50 seconds, 5260/sec Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140322-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 2735 records (2735 ignored) in 0.50 seconds Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140323-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 3198 records (3198 ignored) in 0.00 seconds Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140324-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 5235 records (5235 ignored) in 0.50 seconds Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140325-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 4491 records (4491 ignored) in 0.50 seconds Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140326-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 5275 records (5275 ignored) in 0.50 seconds Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140327-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 6767 records (6767 ignored) in 0.50 seconds Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140328-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 4445 records (4445 ignored) in 0.50 seconds Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140329-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 4031 records (4031 ignored) in 0.50 seconds Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140330-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 6903 records (6903 ignored) in 1.00 seconds, 6903/sec Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140331-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 4958 records (4958 ignored) in 0.50 seconds Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140401-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 4020 records (4020 ignored) in 0.50 seconds Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140402-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 6934 records (6934 ignored) in 1.00 seconds, 6934/sec Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140403-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current Warning: Truncating oversized hostname Skipping bad record (1) 7421 records (7420 ignored, 1 bad) in 0.50 seconds Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140404-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 5571 records (5571 ignored) in 0.50 seconds Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140405-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current Warning: Truncating oversized hostname Skipping bad record (1) 2948 records (2947 ignored, 1 bad) in 0.50 seconds Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140406-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 2848 records (2848 ignored) in 0.50 seconds Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140407-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 4434 records (4434 ignored) in 0.50 seconds Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140408-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 5291 records (5291 ignored) in 0.50 seconds Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140409-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 4572 records (4572 ignored) in 0.00 seconds Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140410-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 8973 records (8973 ignored) in 1.00 seconds, 8973/sec Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140411-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current 3764 records (3764 ignored) in 0.50 seconds Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140413-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current Saving current run data... [04/13/2014 23:48:15] Generating report for April 2014 Generating summary report Saving history information... 2118 records (133 ignored) in 2.50 seconds, 847/sec Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile 20140414-access.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'sunpin' Reading history file... webalizer.hist Reading previous run data.. webalizer.current Saving current run data... [04/14/2014 22:00:01] Generating report for April 2014 Generating summary report Saving history information... 2759 records in 2.00 seconds, 1379/sec Going to bed now. I will see what it did be in the morning.
Except that you know that's not true. I got all my months back. Somehow. I'm not sure my script did anything because April 1-5 is missing and six through 13 has garbage data. Any ideas?
The files in that directory do not contain the log data of your websites, so they can not be used to restore log data for sites. The log data of the websites is in the log directory of the sites which is a bind mount to /var/log/ispconfig/httpd/....
I have the log data of my websites as well. Right now, I'm up to April be in the remaining month to correct.
Merged my April logs to create a log file called restore.log. Deleted webalizer.current from my stats directory Ran webalizer -n restore.log. No effect. Code: root@rule303:/usr/src/pyscripts# python mergeLogs.py root@rule303:/usr/src/pyscripts# cd /var/www/clients/client5/web11/log root@rule303:/var/www/clients/client5/web11/log# webalizer -n restore.log Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile /var/log/apache2/access.log.1 (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'restore.log' Reading history file... /var/www/clients/client5/web11/web/stats/webalizer.hist Reading previous run data.. /var/www/clients/client5/web11/web/stats/webalizer.current 3268 records (3268 ignored) in 0.00 seconds root@rule303:/var/www/clients/client5/web11/log# webalizer -n restore.log Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile /var/log/apache2/access.log.1 (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'restore.log' Reading history file... /var/www/clients/client5/web11/web/stats/webalizer.hist Previous run data not found... Saving current run data... [04/13/2014 06:45:01] Generating report for April 2014 Generating summary report Saving history information... 3268 records in 1.00 seconds, 3268/sec root@rule303:/var/www/clients/client5/web11/log# Getting the rest of the history back has saved my job for a moment. Need to get April squared away.
You might want to take a look at the ispconfig script /usr/local/ispconfig/server/cron_daily.php to see which parameters are used by ispconfig to generate the statistics. ispconfig geerates the statistics day by day, so you might use a script that loops trough the log files aftre you uncompressed them.
Deleted current. Got a clean response on the following: Code: root@rule303:/var/www/clients/client5/web11/log# webalizer restore.log Webalizer V2.01-10 (Linux 3.8.0-29-generic) locale: en_US.UTF-8 Using logfile restore.log (clf) Using default GeoIP database Creating output in /var/www/clients/client5/web11/web/stats Hostname for reports is 'ellatinodigital.com' Reading history file... /var/www/clients/client5/web11/web/stats/webalizer.hist Previous run data not found... Generating report for March 2014 Saving current run data... [04/15/2014 11:17:45] Generating report for April 2014 Generating summary report Saving history information... 64391 records in 22.00 seconds, 2926/sec What did you want me to see in the mail log daily? This is my Python code: Code: #!/usr/bin/env python # -*- coding: utf-8 -*- #Copyright 2014 Christopher James Uzal. #This will merge the log file to restore the statistics. import sys, os path = r'/var/www/clients/client5/web11/log/' logs = [] os.chdir(path) #Get the filenames into a list logs += [each for each in os.listdir(path) if each.endswith('-access.log')] logs.sort() os.system('mergelog ' + ' '.join(logs) + ' > restore.log ' )
the code and webalizer that ispcpnfig uses to create the statistics for all sites: Code: $sql = "SELECT domain_id, domain, document_root, web_folder, type, parent_domain_id FROM web_domain WHERE (type = 'vhost' or type = 'vhostsubdomain') and stats_type = 'webalizer' AND server_id = ".$conf['server_id']; $records = $app->db->queryAllRecords($sql); foreach($records as $rec) { //$yesterday = date('Ymd',time() - 86400); $yesterday = date('Ymd', strtotime("-1 day", time())); $log_folder = 'log'; if($rec['type'] == 'vhostsubdomain') { $tmp = $app->db->queryOneRecord('SELECT `domain` FROM web_domain WHERE domain_id = '.intval($rec['parent_domain_id'])); $subdomain_host = preg_replace('/^(.*)\.' . preg_quote($tmp['domain'], '/') . '$/', '$1', $rec['domain']); if($subdomain_host == '') $subdomain_host = 'web'.$rec['domain_id']; $log_folder .= '/' . $subdomain_host; unset($tmp); } $logfile = escapeshellcmd($rec['document_root'].'/' . $log_folder . '/'.$yesterday.'-access.log'); if(!@is_file($logfile)) { $logfile = escapeshellcmd($rec['document_root'].'/' . $log_folder . '/'.$yesterday.'-access.log.gz'); if(!@is_file($logfile)) { continue; } } $domain = escapeshellcmd($rec['domain']); $statsdir = escapeshellcmd($rec['document_root'].'/'.($rec['type'] == 'vhostsubdomain' ? $rec['web_folder'] : 'web').'/stats'); $webalizer = '/usr/bin/webalizer'; $webalizer_conf_main = '/etc/webalizer/webalizer.conf'; $webalizer_conf = escapeshellcmd($rec['document_root'].'/log/webalizer.conf'); if(is_file($statsdir.'/index.php')) unlink($statsdir.'/index.php'); if(!@is_file($webalizer_conf)) { copy($webalizer_conf_main, $webalizer_conf); } if(@is_file($webalizer_conf)) { setConfigVar($webalizer_conf, 'Incremental', 'yes'); setConfigVar($webalizer_conf, 'IncrementalName', $statsdir.'/webalizer.current'); setConfigVar($webalizer_conf, 'HistoryName', $statsdir.'/webalizer.hist'); } if(!@is_dir($statsdir)) mkdir($statsdir); exec("$webalizer -c $webalizer_conf -n $domain -s $domain -r $domain -q -T -p -o $statsdir $logfile"); }
Solved. Definitely need to merge the log files before restoring. Going one file at a time doesn't work from my little adventure but I could be wrong. Also, you need to delete the webalizer.current file and pop out the current day from your list (or array) and then, in spite of warnings in the documentation, you can pull this off in the middle of the month. w00t. Thanks for reading.
