Below is a cut off, of the default cron file of ISPConfig 3. Till, can we use it to update awstats, as frequently as we want? My replacements: - I removed every logrotate action you had and everything else, that had to do with modifying files - I erased every yesterday variable and I used the default access.log for each website - I tested it. As I see after the tests, when I run it , then the stats REPLACED and updated for today. When the real cron runs every night, then the stats created with the above script, replaced again bringing us back to the state that ISPConfig wants. If (after 00:30) I run the script again, then the stats replaced again, but actually only the 'new' day is modified. I think it works. Can I have your opinion? Code: <?php require('/usr/local/ispconfig/server/lib/config.inc.php'); require('/usr/local/ispconfig/server/lib/app.inc.php'); set_time_limit(0); // make sure server_id is always an int $conf['server_id'] = intval($conf['server_id']); // Load required base-classes $app->uses('ini_parser,file,services,getconf'); ####################################################################################################### // Create awstats statistics ####################################################################################################### $sql = "SELECT domain_id, domain, document_root FROM web_domain WHERE stats_type = 'awstats' AND server_id = ".$conf['server_id']; $records = $app->db->queryAllRecords($sql); $web_config = $app->getconf->get_server_config($conf['server_id'], 'web'); foreach($records as $rec) { $logfile = escapeshellcmd($rec['document_root'].'/log/access.log'); $domain = escapeshellcmd($rec['domain']); $statsdir = escapeshellcmd($rec['document_root'].'/web/stats'); $awstats_pl = $web_config['awstats_pl']; $awstats_buildstaticpages_pl = $web_config['awstats_buildstaticpages_pl']; $awstats_conf_dir = $web_config['awstats_conf_dir']; $awstats_website_conf_file = $web_config['awstats_conf_dir'].'/awstats.'.$domain.'.conf'; if(is_file($awstats_website_conf_file)) unlink($awstats_website_conf_file); if(!is_file($awstats_website_conf_file)) { $awstats_conf_file_content = 'Include "'.$awstats_conf_dir.'/awstats.conf" LogFile="/var/log/ispconfig/httpd/'.$domain.'/access.log" SiteDomain="'.$domain.'" HostAliases="www.'.$domain.' localhost 127.0.0.1"'; file_put_contents($awstats_website_conf_file,$awstats_conf_file_content); } if(!@is_dir($statsdir)) mkdir($statsdir); // awstats_buildstaticpages.pl -update -config=mydomain.com -lang=en -dir=/var/www/domain.com/web/stats -awstatsprog=/path/to/awstats.pl $command = "$awstats_buildstaticpages_pl -update -config='$domain' -lang=en -dir='$statsdir' -awstatsprog='$awstats_pl'"; if($awstats_pl != '' && $awstats_buildstaticpages_pl != '' && fileowner($awstats_pl) == 0 && fileowner($awstats_buildstaticpages_pl) == 0) { exec($command); rename($rec['document_root'].'/web/stats/awstats.'.$domain.'.html',$rec['document_root'].'/web/stats/index.html'); $app->log('Created awstats statistics with command: '.$command,LOGLEVEL_DEBUG); } else { $app->log("No awstats statistics created. Either $awstats_pl or $awstats_buildstaticpages_pl is not owned by root user.",LOGLEVEL_WARN); } } die("finished.\n"); ?> I named the file as /usr/local/ispconfig/server/mycron.php and I can run it as Code: /usr/bin/php -q /usr/local/ispconfig/server/mycron.php
Ok. I think I confirmed it. I run it for about 2-3 days now and it works! I can update awstats anytime I want (e.g. through cron) By the way: I can have Greek (that means ... any language that awstats support) just by changing the line: Code: $command = "$awstats_buildstaticpages_pl -update -config='$domain' -lang=en -dir='$statsdir' -awstatsprog='$awstats_pl'"; with this one Code: $command = "$awstats_buildstaticpages_pl -update -config='$domain' -lang=[COLOR="Red"]gr[/COLOR] -dir='$statsdir' -awstatsprog='$awstats_pl'"; Actually, I have to run it, once, after the default cron (e.g. 00:40). That means, that I can have Greek and frequently updated stats WITHOUT even change a single line of the ISPConfig php files. A suggestion: Maybe ISPConfig could have an implementation like this. The default cron.daily could call a script to update stats and after that make the necessary rotations. With this, we could call the script anytime we want. Even through ISPConfig interface via a button.
The previous code applies only up to ISPConfig 3.0.3. For ISPConfig 3.0.4 I updated the code in the "Extending perfect server" tutorial. I did it today, so I have not test it enough.
I re-updated the tutorial with the following code Code: <?php /* Copyright (c) 2007, Till Brehm, projektfarm Gmbh All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of ISPConfig nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ require('/usr/local/ispconfig/server/lib/config.inc.php'); require('/usr/local/ispconfig/server/lib/app.inc.php'); set_time_limit(0); // make sure server_id is always an int $conf['server_id'] = intval($conf['server_id']); // Load required base-classes $app->uses('ini_parser,file,services,getconf'); ####################################################################################################### // Create awstats statistics ####################################################################################################### $sql = "SELECT domain_id, domain, document_root FROM web_domain WHERE stats_type = 'awstats' AND server_id = ".$conf['server_id']; $records = $app->db->queryAllRecords($sql); $web_config = $app->getconf->get_server_config($conf['server_id'], 'web'); foreach($records as $rec) { $logfile = escapeshellcmd($rec['document_root'].'/log/access.log'); $domain = escapeshellcmd($rec['domain']); $statsdir = escapeshellcmd($rec['document_root'].'/web/stats'); $awstats_pl = $web_config['awstats_pl']; $awstats_buildstaticpages_pl = $web_config['awstats_buildstaticpages_pl']; $awstats_conf_dir = $web_config['awstats_conf_dir']; $awstats_website_conf_file = $web_config['awstats_conf_dir'].'/awstats.'.$domain.'.conf'; if(is_file($awstats_website_conf_file)) unlink($awstats_website_conf_file); $sql = "SELECT domain FROM web_domain WHERE (type = 'alias' OR AND type = 'subdomain') server_id = ".$conf['server_id']; $aliases = $app->db->queryAllRecords($sql); $aliasdomain = ''; if(is_array($aliases)) { foreach ($aliases as $alias) { $aliasdomain.= ' '.$alias['domain']. ' www.'.$alias['domain']; } } if(!is_file($awstats_website_conf_file)) { $awstats_conf_file_content = 'Include "'.$awstats_conf_dir.'/awstats.conf" LogFile="/var/log/ispconfig/httpd/'.$domain.'/access.log" SiteDomain="'.$domain.'" HostAliases="www.'.$domain.' localhost 127.0.0.1"'.$aliasdomain; file_put_contents($awstats_website_conf_file,$awstats_conf_file_content); } if(!@is_dir($statsdir)) mkdir($statsdir); if (date("d") == 1) { $awmonth = date("m")-1; $awyear = date("Y"); if (date("m") == 1) { $awyear = date("Y")-1; $awmonth = "12"; } $statsdirold = $statsdir."/".$awyear."-".$awmonth."/"; if (!is_dir($statsdirold)) { mkdir($statsdirold); $command = "$awstats_buildstaticpages_pl -month='$awmonth' -year='$awyear' -update -config='$domain' -lang=".$conf['language']." -dir='$statsdir' -awstatsprog='$awstats_pl'"; if($awstats_pl != '' && $awstats_buildstaticpages_pl != '' && fileowner($awstats_pl) == 0 && fileowner($awstats_buildstaticpages_pl) == 0) { exec($command); if(is_file($rec['document_root'].'/web/stats/index.html')) unlink($rec['document_root'].'/web/stats/index.html'); rename($rec['document_root'].'/web/stats/awstats.'.$domain.'.html',$rec['document_root'].'/web/stats/awsindex.html'); if(!is_file($rec['document_root']."/web/stats/index.php")) copy("/usr/local/ispconfig/server/conf/awstats_index.php.master",$rec['document_root']."/web/stats/index.php"); $app->log('Created awstats statistics with command: '.$command,LOGLEVEL_DEBUG); } else { $app->log("No awstats statistics created. Either $awstats_pl or $awstats_buildstaticpages_pl is not owned by root user.",LOGLEVEL_WARN); } $files = scandir($statsdir); foreach ($files as $file) { if (substr($file,0,1) != "." && !is_dir($file) && substr($file,0,1) != "w" && substr($file,0,1) != "i") copy("$statsdir"."/"."$file","$statsdirold"."$file"); } } } $awmonth = date("n"); $awyear = date("Y"); $command = "$awstats_buildstaticpages_pl -month='$awmonth' -year='$awyear' -update -config='$domain' -lang=".$conf['language']." -dir='$statsdir' -awstatsprog='$awstats_pl'"; if($awstats_pl != '' && $awstats_buildstaticpages_pl != '' && fileowner($awstats_pl) == 0 && fileowner($awstats_buildstaticpages_pl) == 0) { exec($command); if(is_file($rec['document_root'].'/web/stats/index.html')) unlink($rec['document_root'].'/web/stats/index.html'); rename($rec['document_root'].'/web/stats/awstats.'.$domain.'.html',$rec['document_root'].'/web/stats/awsindex.html'); if(!is_file($rec['document_root']."/web/stats/index.php")) copy("/usr/local/ispconfig/server/conf/awstats_index.php.master",$rec['document_root']."/web/stats/index.php"); $app->log('Created awstats statistics with command: '.$command,LOGLEVEL_DEBUG); } else { $app->log("No awstats statistics created. Either $awstats_pl or $awstats_buildstaticpages_pl is not owned by root user.",LOGLEVEL_WARN); } } die("finished.\n"); ?> I changed a lot the script (e.g. I removed the check for the second day of month). I tried it and it works. Can anyone confirm it? Till, I would appreciate it a lot, if you could take a look. The concept is: - If this is the first day of the month, go back one day and create the old stats and copy the previous month if only, the dir (year-month) does not exist, else do nothing. (This part will execute only once, and will not execute again if the folder will be created) - Make the date correct - Recreate the stats for today (this will execute as many times as the cron will calls this sript)
One more thing. The command : Code: $command = "$awstats_buildstaticpages_pl -month='$awmonth' -year='$awyear' -update -config='$domain' -lang=".$conf['language']." -dir='$statsdir' -awstatsprog='$awstats_pl'"; does not work for my language (Greek). So I have to change -lang=".$conf['language']." with -lang=gr I assume this happens because the awstats knows Greek as "gr" and ISPConfig as "el"
With the new year I had the confirmation that I wanted. The stats updated normally. The only problem, is that I could not see the update. But the update was there! The problem was the lines: Code: if (date("d") == 1) { $month = date("m")-1; if (date("m") == 1) { $year = date("Y")-1; $month = "12"; } } in the file ......stats/index.php Anyway. The stats dissappeared for only one day. When the day was 2 the stats appeared normal. A quick fix is to comment out every instance of the above code in every index.php of every site. Another fix (is a bad fix because we have to mess up with core) is to comment out the above lines in: /usr/local/ispconfig/server/conf/awstats_index.php.master I did the second. The bad thing is that I must remember this when the new update of ISPConfig will be ready.
in mycron.php the lines Code: if (date("m") == 1) { $awyear = date("Y")-1; $awmonth = "12"; } must be replaced with those lines Code: if (date("n") == 0) { $awyear = date("Y")-1; $awmonth = "12"; }
New code. Works good. Eliminates the problem of overwritten stats of previously month, the second day of the current month. The logic, of the creation of awstats of the default cron, is that the second day of the month, the currently stats, are written in a new folder, repressenting the previously month. This overwrites the stats already written in this folder by this script. The new version corrects this. One new addition, is the ability to recreate the stats from ANY month, as log as the data are in /var/lib/awstats or /var/cache/awstats. Using this new feature -first, you can correct any mistakes made by this script in the past and -second, you can add ALL previously stats in the drop down menu. Name it as: /usr/local/ispconfig/server/mycron.php Execute it: Code: /usr/bin/php /usr/local/ispconfig/server/mycron.php Here is the script. Code: <?php /* Copyright (c) 2007, Till Brehm, projektfarm Gmbh All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of ISPConfig nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ require('/usr/local/ispconfig/server/lib/config.inc.php'); require('/usr/local/ispconfig/server/lib/app.inc.php'); set_time_limit(0); // make sure server_id is always an int $conf['server_id'] = intval($conf['server_id']); // Load required base-classes $app->uses('ini_parser,file,services,getconf'); ################################################################ ######################### RECREATE STATS ########################## $redo = 0; // set this to 1 if you want to recreate statistics for a specific month $REMONTH = 11; // select the month $REYEAR = 2011; //select the year $LANG = 'en'; // set your language # After setting the above rerun this script # e.g # php /path/to ################################################################ ################################################################ ################################################################ // Create awstats statistics ################################################################ $sql = "SELECT domain_id, domain, document_root FROM web_domain WHERE stats_type = 'awstats' AND server_id = ".$conf['server_id']; $records = $app->db->queryAllRecords($sql); $web_config = $app->getconf->get_server_config($conf['server_id'], 'web'); foreach($records as $rec) { $logfile = escapeshellcmd($rec['document_root'].'/log/access.log'); $domain = escapeshellcmd($rec['domain']); $statsdir = escapeshellcmd($rec['document_root'].'/web/stats'); $awstats_pl = $web_config['awstats_pl']; $awstats_buildstaticpages_pl = $web_config['awstats_buildstaticpages_pl']; $awstats_conf_dir = $web_config['awstats_conf_dir']; $awstats_website_conf_file = $web_config['awstats_conf_dir'].'/awstats.'.$domain.'.conf'; if(is_file($awstats_website_conf_file)) unlink($awstats_website_conf_file); $sql = "SELECT domain FROM web_domain WHERE (type = 'alias' OR AND type = 'subdomain') server_id = ".$conf['server_id']; $aliases = $app->db->queryAllRecords($sql); $aliasdomain = ''; if(is_array($aliases)) { foreach ($aliases as $alias) { $aliasdomain.= ' '.$alias['domain']. ' www.'.$alias['domain']; } } if(!is_file($awstats_website_conf_file)) { $awstats_conf_file_content = 'Include "'.$awstats_conf_dir.'/awstats.conf" LogFile="/var/log/ispconfig/httpd/'.$domain.'/access.log" SiteDomain="'.$domain.'" HostAliases="www.'.$domain.' localhost 127.0.0.1"'.$aliasdomain; file_put_contents($awstats_website_conf_file,$awstats_conf_file_content); } if(!@is_dir($statsdir)) mkdir($statsdir); if ( $redo == 1 ) { $awmonth = $REMONTH; $awyear = $REYEAR; $statsdirold = $statsdir."/".$awyear."-".$awmonth."/"; if (!is_dir($statsdirold)) { mkdir($statsdirold); } $command = "$awstats_buildstaticpages_pl -month='$awmonth' -year='$awyear' -update -config='$domain' -lang='$LANG' -dir='$statsdirold' -awstatsprog='$awstats_pl'"; if($awstats_pl != '' && $awstats_buildstaticpages_pl != '' && fileowner($awstats_pl) == 0 && fileowner($awstats_buildstaticpages_pl) == 0) { exec($command); rename($statsdirold.'awstats.'.$domain.'.html',$statsdirold.'awsindex.html'); $app->log('Created awstats statistics with command: '.$command,LOGLEVEL_DEBUG); } else { $app->log("No awstats statistics created. Either $awstats_pl or $awstats_buildstaticpages_pl is not owned by root user.",LOGLEVEL_WARN); } } else { if (date("d") == 1) { $awmonth = date("n")-1; $awyear = date("Y"); if (date("n") == 0) { $awyear = date("Y")-1; $awmonth = "12"; } $statsdirold = $statsdir."/".$awyear."-".$awmonth."/"; if (!is_dir($statsdirold)) { mkdir($statsdirold); $command = "$awstats_buildstaticpages_pl -month='$awmonth' -year='$awyear' -update -config='$domain' -lang='$LANG' -dir='$statsdirold' -awstatsprog='$awstats_pl'"; if($awstats_pl != '' && $awstats_buildstaticpages_pl != '' && fileowner($awstats_pl) == 0 && fileowner($awstats_buildstaticpages_pl) == 0) { exec($command); rename($statsdirold.'awstats.'.$domain.'.html',$statsdirold.'awsindex.html'); $app->log('Created awstats statistics with command: '.$command,LOGLEVEL_DEBUG); } else { $app->log("No awstats statistics created. Either $awstats_pl or $awstats_buildstaticpages_pl is not owned by root user.",LOGLEVEL_WARN); } } } // ispconfig, on the second day of month, overwrites the stats created before by this script // to overwrite ispconfig default execution on 2nd day of month we must execute the below code if (date("d") == 2) { $awmonth = date("n")-1; $awyear = date("Y"); if (date("n") == 0) { $awyear = date("Y")-1; $awmonth = "12"; } $statsdirold = $statsdir."/".$awyear."-".$awmonth."/"; $command = "$awstats_buildstaticpages_pl -month='$awmonth' -year='$awyear' -update -config='$domain' -lang='$LANG' -dir='$statsdirold' -awstatsprog='$awstats_pl'"; if($awstats_pl != '' && $awstats_buildstaticpages_pl != '' && fileowner($awstats_pl) == 0 && fileowner($awstats_buildstaticpages_pl) == 0) { exec($command); rename($statsdirold.'awstats.'.$domain.'.html',$statsdirold.'awsindex.html'); $app->log('Created awstats statistics with command: '.$command,LOGLEVEL_DEBUG); } else { $app->log("No awstats statistics created. Either $awstats_pl or $awstats_buildstaticpages_pl is not owned by root user.",LOGLEVEL_WARN); } } // end of overwrite // unfortunately the above will execute many times (as many as you run this script by cron) $awmonth = date("n"); $awyear = date("Y"); $command = "$awstats_buildstaticpages_pl -month='$awmonth' -year='$awyear' -update -config='$domain' -lang='$LANG' -dir='$statsdir' -awstatsprog='$awstats_pl'"; if($awstats_pl != '' && $awstats_buildstaticpages_pl != '' && fileowner($awstats_pl) == 0 && fileowner($awstats_buildstaticpages_pl) == 0) { exec($command); if(is_file($rec['document_root'].'/web/stats/index.html')) unlink($rec['document_root'].'/web/stats/index.html'); rename($rec['document_root'].'/web/stats/awstats.'.$domain.'.html',$rec['document_root'].'/web/stats/awsindex.html'); if(!is_file($rec['document_root']."/web/stats/index.php")) copy("/usr/local/ispconfig/server/conf/awstats_index.php.master",$rec['document_root']."/web/stats/index.php"); $app->log('Created awstats statistics with command: '.$command,LOGLEVEL_DEBUG); } else { $app->log("No awstats statistics created. Either $awstats_pl or $awstats_buildstaticpages_pl is not owned by root user.",LOGLEVEL_WARN); } } //end redo } // end foreach die("finished.\n"); ?> To run it for a specific month, edit the variables $redo, $REMONTH, $REYEAR, $LANG in the beggining of the script If the $redo variable is 1, the script will ONLY recreate the month declared in $REMONTH, of the year $REYEAR, in you preffered language (declared in $LANG) If the $redo variable is 0, the script will run normally. It will create the stats up to the time executed, for the current month, eliminating overwrites b ispconfig.
Just checked the code after real operations. The operations are: - my script run at 23:55, so I had a full February report - ISPConfig cron run at 00:30, so the normal operations took place - my script run again at 00:40 Things are almost normal. The default page shows March statistics, but the drop down menu has 2012-2 checked. In fact there are 2 instances of 2012-2. The First is at the top of the list, representing the real February, and the Second is at the bottom of the drop down list, representing March. The problem (which is not a real problem) is related with the code : Code: if (date("d") == 1) { $month = date("m")-1; if (date("m") == 1) { $year = date("Y")-1; $month = "12"; } } in index.php The above code, works normal for ISConfig's scheduled cron job, but has this effect in my script. Anyway, I think that tomorrow, (which is not -- date("d") == 1 --) the list will be ok. Remove the above lines, breaks the list for ISPConfig, and corrects it for my script. I 'll stick with the ISPConfig implementation, as I do not like to mess with core. After all, one day with a little mistake in the list, is not a big deal
Hmm, to be more specific, certain days aren't generated, the last days of august and the first of september...
Do you use the latest file I upload? As you will see in my previous posts, there is a problem only on the first day. Actually, it's not a problem, just a wrong item in the list, but only for the first day. p.s. I use the same script, with no problems.
Yes I do use the uploaded file. The weird thing is that, even with REDO=1, the latest days of august are not 'statistified', but the access logs are there and the permissions are in order... so that's why I'm unsure about the reason why it is not regenerating... any suggestion?
I just retest the script and it works ok on me. Let the default cron.php and mycron.php run tonight, and see if tomorrow is everything ok. 1) Do you use the latest ISPConfig? 2) Is the issue happening on all sites? 3) Did the issue happened before?