ISPConfig 3, Dovecot 2 and traffic accounting

Hi everyone at HowToForge,

this is my first post here.

I'd like to use ISPConfig 3 with Dovecot (preferrably v2.x) and do IMAP/POP3/submission traffic accounting. Preferrably on CentOS 6, but Ubuntu 12.04 LTS or Debian 6 would also work. According to a recent forum post by Till this should be possible with ISPConfig 3.0.4.6.

I'd like to give this a try and set this up in a VM tomorrow. Before I do, is there something I should be aware of in this context? Are there any known issues with this setup? Should I prefer Debian 6 over Ubuntu 12.04? I will probably use this how-to unless you recommend otherwise.

And most importantly, should I expect traffic accounting to work out of the box, or will I need to make any manual changes to make it work?

Thanks in advance!

 

No.

Both Debian and Ubuntu are fine, but Debian is a bit more stable.

It should work out of the box.

 

"Should" or is working in 3.0.4.6? . This feature doesn't work for me in 12.04 LTS + dovecot 2

Any way to check it? Is working for you?

 

So my installation is done, following Falko's great how-to to the point.

I've created a mail domain, and a mail user on this domain. I've used an e-mail client on another computer to connect to this servers' IMAP and Submission ports, and have sent and received e-mail, amounting to several megabytes of mail traffic on both protocols / ports.

Then I checked the mail traffic statistics for this user on ISPConfig's web interface. It listed 0. Which is understandable, since, most likely, these statistics are not updated real-time but by a regularly running cronjob.

So I looked at the newly configured system and found (running

Code:

crontab -l

as root) a daily (or rather nightly, running at 00:30 by default) ISPConfig cron job which runs this file:

Code:

/usr/local/ispconfig/server/cron_daily.php

This PHP script imports the servers' ISPConfig related configuration and some helper classes (such as for structured databases access). After that, it contains multiple sections which each start with a comment. The first such section is

Code:

// store the mailbox statistics in the database

The code in this section searches all mail users' maildir locations for a file called

Code:

ispconfig_mailsize

Example for mail account [email protected]:

Code:

/var/vmail/domain.tld/someuser/ispconfig_mailsize

So for each user, if ispconfig_mailsize exists, it renames it to "ispconfig_mailsize_save" then reads the file line by line, type casting each line to integer, and summing them up. It then add this value to any traffic previously recorded for this user (and stored in the mysql database, in table "mail_traffic") during the running month, if any, and stores this new value in the database

For each user where ispconfig_mailsize does not exist, nothing is done.

In my case, no such file exists (after sending large e-mails with file attachments in and out).

There are other sections in this file, labelled as follows:

Code:

// Create webalizer statistics
// Create awstats statistics
// Make the web logfiles directories world readable to enable ftp access
// Manage and compress web logfiles and create traffic statistics
// Rotate the ispconfig.log file
// Cleanup website tmp directories
// Cleanup logs in master database (only the "master-server")
// enforce traffic quota (run only on the "master-server")
// deactivate virtual servers (run only on the "master-server")
// Create website backups

None of these seem to refer to mail traffic accounting.

So what's the mechanism which creates the "ispconfig_mailsize" files in the first place?

Code:

# rgrep 'ispconfig_mailsize' /usr/local/
/usr/local/ispconfig/server/plugins-available/mail_plugin.inc.php:			// if(is_file($data['old']['maildir'].'.ispconfig_mailsize'))exec('mv -f '.escapeshellcmd($data['old']['maildir']).'.ispconfig_mailsize '.escapeshellcmd($data['new']['maildir']));
/usr/local/ispconfig/server/cron_daily.php:	if(@is_file($rec['maildir'].'/ispconfig_mailsize')) {
/usr/local/ispconfig/server/cron_daily.php:		rename($rec['maildir'].'/ispconfig_mailsize',$rec['maildir'].'/ispconfig_mailsize_save');
/usr/local/ispconfig/server/cron_daily.php:		$lines = file($rec['maildir'].'/ispconfig_mailsize_save');
/usr/local/ispconfig/server/cron_daily.php:		if(@is_file($rec['maildir'].'/ispconfig_mailsize_save')) unlink($rec['maildir'].'/ispconfig_mailsize_save');
/usr/local/ispconfig/server/plugins-enabled/mail_plugin.inc.php:			// if(is_file($data['old']['maildir'].'.ispconfig_mailsize'))exec('mv -f '.escapeshellcmd($data['old']['maildir']).'.ispconfig_mailsize '.escapeshellcmd($data['new']['maildir']));
#

/usr/local/ispconfig/server/plugins-available/mail_plugin.inc.php is a symbolic link to /usr/local/ispconfig/server/plugins-enabled/mail_plugin.inc.php which only contains one reference to this filename, but the line is commented out.
/usr/local/ispconfig/server/cron_daily.php is the file we already discussed above, it does not create these files either.

Maybe the file is created by Dovecot or some other daemon, based on modifications ISPConfig may have made to their configuration files during installation?

Code:

# rgrep 'ispconfig_mailsize' /etc/
grep: /etc/fonts/conf.d/30-defoma.conf: No such file or directory
#

Apparently that's not the case either. Maybe it's somewhere next to the mail storage:

Code:

# rgrep 'ispconfig_mailsize' /var/vmail/
/var/vmail/.mailfilter:`echo $SIZE >> /var/vmail/$HOST/$USER/ispconfig_mailsize`
#

Aha, so this is something. So some searching on the web will tell you that maildrop is a mail filter/mail delivery agent, which comes with the Courier MTA by default, but can also be used standalone, as well as in combination with Dovecot:
http://www.courier-mta.org/maildrop/
http://www.dovecot.org/list/dovecot/2011-February/057195.html

Is maildrop packaged for Debian 6?

Code:

# apt-cache search maildrop
courier-maildrop - Courier mail server - mail delivery agent
getmail4 - mail retriever with support for POP3, IMAP4 and SDPS
maildrop - mail delivery agent with filtering abilities
rdeliver - A fully functioning mail filter with RubyFilter
sensible-mda - Mail Delivery Agent wrapper
spamprobe - Bayesian spam filter
#

OK, there are both the Courier one and a standalone maildrop package. Are any of these installed (and hopefully configured, too) now that I followed Falko's how-to?

Code:

# dpkg -l maildrop courier-maildrop
No packages found matching maildrop.
No packages found matching courier-maildrop.
#

Hmm, no. Maybe another installed package provides the "maildrop" command?

Code:

# which maildrop
#

Apparently not.

So I'm not sure how mail (IMAP4r2, POP3, SMTP via Submission port) traffic accounting could work with this setup. Could you please elaborate, Falko? Did I miss something obvious (or not so obvious)?

Thank you very much in advance.

 

It would be great to get a confirmation that my suspicion (traffic accounting with Dovecot 2 is not currently implemented) is correct. If it is, I could possibly spend some time on looking into how it could be implemented and provide some feedback.

 

AFAIK this is correct. I can't tell if this is part of ISPConfig 3.0.5 or not. Will have to ask Till.

 

Thanks, Falko. Have you two had a chance to discuss this, yet?

 

Thanks for your reply, Till.

My understanding is that there is a standalone implementation available of maildrop which can also be integrated between MTA and dovecot LDA. It probably makes more sense to work with sieve filters, though.

About ispconfig_mailsize, I was thinking of *somehow* making dovecot (or something added to dovecot) write out (properly formatted) ispconfig_mailsize files, too, so that the ISPConfig script doesn't need to be expanded/modified and run another 'find' on the file system, slowing down the system unnecessarily during the nightly cron job. Plus, that would mean less code to maintain on the ISPConfig side once it works. On the other hand it might require in a more complex sieve filter / maildrop setup (which could also increase processing overhead per e-mail, and that would be worse than a second 'find').

My understanding here is that, while, as you're saying, standard sieve does not allow for writing files to the file system (nor executing external applications which could do so), there are extensions to the sieve filtering language, written and documented by dovecot developers as part of the Piegeonhole project, which allow you to pipe a copy of an e-mail into an external command. Now if this external command is a wrapper around 'wc -c' and writes its output to disk, you have a simplistic implementation for counting mail traffic with little overhead. If needed (such as for only counting the message body), one could use a more intelligent wrapper there, such as one written in PHP.

http://wiki2.dovecot.org/Pigeonhole
http://wiki2.dovecot.org/Pigeonhole/Sieve
http://wiki2.dovecot.org/Pigeonhole/Sieve/Plugins/Extprograms

So I think this should be possible already now. Of course, the plain size of e-mails transferred across IMAP/POP3 differs from the real traffic you're causing there, but I would assume it's close enough (and better than nothing). To get somewhat close to a proper implementation of mail traffic accounting you would also want/need to measure SMTP e-mail traffic. My current understanding is that this is not currently implemented in ISPConfig for any supported MTA (please correct me if wrong).

So I also looked at how this could possibly be done with Postfix. While there can be 'size=...' values in your ESMTP log, this turns out to be an optional flag, i.e. clients/MTAs are not required to send it (and I think most clients actually don't). So the friendly people in #postfix on FreeNode recommended against relying on it and suggested to use a policy (with Postfix policy daemon) instead, because the 'size' parameter is always transferred there.

Here's an incomplete and untested proof of concept shell script which should work for this purpose, logging the sizes of all e-mail it learns about to disk.

Code:

#!/bin/sh
# http://www.postfix.org/SMTPD_POLICY_README.html
IFSBACKUP=$IFS
IFS=`echo -e "=\r\n"`
while read param value morevalues
do
  if [ "$morevalues" != '' ]
  then
    echo 'Error: More input found than expected' >&2
    exit 1
  fi
  case $param in
    sasl_username )
      domain=`echo $value | cut -d'@' -f2`
      user=`echo $value | cut -d'@' -f1`
      ;;
    size )
      size= $value
      ;;
  esac
done
IFS=$IFSBACKUP

echo -e 'action=dunno\n\n'

echo "$size" >> /var/vmail/$domain/$user/LOGFILE

This policy should only be used for SMTP mail, and only be applied after successful client authentication has taken place via SASL. This should be possible by placing it at the proper position in the 'restrictions' section of postfix' main.cf

So, while some of this is still theoretical, my general impression is that traffic accounting with all non legacy versions of Postfix and with Dovecot 2 should be achievable. Now "someone" "just" needs to find the time to actually implement and test this. ;-) I can't really promise to do it, though (but will try to find some time).

 

I implemented a mail.log parsing fallback for ISPConfig 3.0.5 that fetches mail sizes from the amavis log lines in the mail.log.
So we finally have a traffic accounting for dovecot in the daily cron.

 

Thank you Croydon!
Have you tried to verify whether Amavis' mail log is a reliable source for this kind of information then? I.e. does it log the size of every e-mail passing through it (and not some other sizes), and do these values look correct?

If this works, we'll only be lacking SMTP traffic accounting, that's quite some progress.

 

The sizes given by amavis seem to be quite reliable and include attachments so it is a good thing to work with I think.
Amavis checks inbound and outbound mails. So the current implementation should show all inbound and outbound traffic for each mailbox just (or nearly) like the traffic accounting in ispconfig for courier does.

 

How do i get mail traffic monitoring to work with the Amavis log - is there a how-to somewhere ?

I have IPSConfig 3.4.6 and dovecot on a centOS 6.3 box.

 

Hi,

Any progress on this? I have 0 mailbox traffic and I am trying to enable it.

Regards,

 

It is working in my setup. I upgraded to version 3.0.5.2 earlier this year and mailbox traffic now works. I dont know how accurate it is though.

My setup is:
Ubuntu Server 13.04
Postfix
Dovecot
ISPConfig 3.0.5.2