I've installed the new 3.1 final and I'm impressed with the amount of updates and features done. Good job! Now, I'd like to know how quotas are managed with Dovecot. I have an existing Dovecot dict-based quota config that I want to integrate ISPC with, and I'd expect ISPC to use the command line tools to get/set/recalc quota, etc. doveadm quota is the tool for it in DOvecot v2+ and I see some places that use it. However I'm seeing weird, probably outdated comments in the sources, see this commit. What is wrong with using doveadm quota? No further info is in the commit nor in code comments. I used to have occasional quota mishaps with mdbox, but not any more and should they ever come up again they can be "fixed" by regular doveadm quota recalc -A commands. BTW, I think the mentioned commit is wrong, since the if uses $retval instead of $tmp_retval and as such it would never set $dovecot to true.
ISPConfig setups are SQL based system and in this setup dovecot, reads the quota directly from the MySQL database. There is no need to use the doveadm command. Take a look at the dovecot sql conf file to see how it works.
Ah, sorry, I mean used storage within the allowed quota. I'm aware where and how ispconfig stores the quota rules, that's fine. What I can't seem to find out how it calculates or gets (from dovecot) the actual storage space used. The most appropriate way would be adopting a fixed running quota db schema OR asking doveadm quota to provide that info. Which is it in case of ispconfig?
You can find the code in the file /usr/local/ispconfig/server/lib/classes/cron.d/100-monitor_email_quota.inc.php The code works fine on all supported perfect server setups.
OK, I see I'm on my own. But could you please answer the other 2 related issues I mentioned: -- Why is using doveadm quota bad (it is according to the mentioned code comment)? In my experience based on about a dozen Dovecot installs from small up to ~2k mailboxes it works just fine. -- Second, the linked code commit is probably wrong, checking the wrong variable. I can issue a ticket if you want, though. I'm an experienced admin and sw dev so I can dig up everything using the source code. But completely ignoring parts of my question is not why I subscribed here. You do a great work developing this free CPL (hence I've chosen it with the thousands of other happy users) and answer pretty quickly, there is no problem at all with that (thanks), but I can't feel I can say you answered the question I asked.
I have not written that comment as you can see in the commit. If there is such a comment, then a specific version of dovecot produced errors or did not work correctly. This might have been fixed by dovecot in the meantime but two years ago there was a problem with using it. Yes, please submit a Bugreport and we will check if there is something wrong there. We support just specific setups with ISPConfig these setups are defined in the perfect server guides. We already support many options and we have to support not just dovecot 2+, there are still many dovecot 1.x setups out there and also courier setups etc, that's why the ispconfig statistics code has several fallbacks to other methods to get the required usage data. We are aware that there are thousands of other possible ways to install a server but we can not support all of them. I have not ignored anything from your question, but as you can see in the bug tracker that I'm not the person who added that commit and if that person (which is an experienced developer and admin btw., so he had his reasons to add that comment) did not add any additional note, how should I be able to answer your question? If I would know the answer, I have posted it. But I do not make posts on everything I don't know, I just answer questions that I have the answer for. If you like to know the exact reasons, send a PM to @florian030 and ask him why he added that comment.
OK, thanks, of course it's reasonable to only support a certain set of configurations. Since I'm always using latest stable versions if possible I tend to forget you need to support all kinds of legacy software too. I think I'll create a custom script out of 100-monitor_email_quota.inc.php for now. Since I think Dovecot has probably the largest user base nowadays, it might be beneficial to fix that calculation method to use doveadm. I'll post an enhancement request in the ticket later, too.
I will take a look at it to see if it works with all supported dovecot versions now. At the moment, ispconfig reads the quota usage on dovecot servers directly from the .quotausage file (where dovecot stores the quota values) which is more efficient than using doveadm as we do not have to invoke a program for each mailbox, so while the doveadm codebranch is not used, the current way to read the quota is probably faster and more efficient than running doveadm. You should compare your dovecot config file with the one from ispconfig and check if you can integrate this into your config: plugin { quota = dict:user::file:/var/vmail/%d/%n/.quotausage sieve=/var/vmail/%d/%n/.sieve sieve_max_redirects = 25 } so ispconfig can read the quota usage directly like it is done in the perfect server setups. Btw. I have a customer which runs an ispconfig mail cluster with more than 15 thousand mailboxes, so the current ispconfig setup which is used there too scales quite well.
Yes, I could certainly implement the ISPC method, but I already have a Dovecot dictionary quota in mysql which is more efficient than file-based storage. If possible I wouldn't change it, though, it tends to work very well. It's probably the easier way for me to change the cron srcipt. It's a "collector type" script anyway that puts recent usage info in a monitoring log table and from there it's gathered when it's needed for displaying and such. OTOH, the current usage monitor/collector script issues a doveadm recalc command before reading .quotausage which kind of makes the point of using efficient quota files moot (running recalc is quite a heavy process when running on many mailboxes). Well, in theory, because that exact part seems to be broken as I mentioned and recalc is never issued if I'm right. Starting doveadm quota get does the same thing but it has the benefit of decoupling the quota calculation and reading mechanism from the ISPC code. It makes a standardised usage of Dovecot native quotas and it becomes possible to choose the quota backend freely. Or, if all supported Dovecot versions at least support DB-based (dictionary or not) quotas, it would be possible to change ISPC to use DB quota backend and read usage info directly from one of its DB tables.
I guess we can remove the recalc anyway as it works without it and the command is not run at the moment or at least we just run it not that often like once a day or once every week. We might be able to alter the code to do a fallback to query the dovecot command for the quota in case that the dovecot command is there but the .quotausage file does not exist, so it should provide a fallback for custom setups like yours that use dovecot but don't have the .quotausage file.
That would be nice. If you run doveadm quota get -A, you get all mailbox quotas to iterate over in a simple loop, and with something like doveadm quota get -u '*@domain.com' it's possible to add some filtering. Well, that assumes you have an iterate_query definition, which IIRC the current ISPC config template doesn't have - though it's very easy to implement, just a single select query and config line, something like: Code: iterate_query = select email from mail_user where access = 'y'
I added a patch with the outlined solutiuon. Is it possible that it gets reviewed and eventually included upstream? What is the normal workflow for it?
Thank you for the patch. I've assigned the feature request to the 3.2 milestone, we will test it and integrate it when it works properly and does not cause any issues with existing installations like old dovecot 1.x versions.
Sorry to be a pain, I'm not quite up to your level and probably haven't followed this correctly but I think my problem is related. I have this awesome 3.1.1 server I'm so happy with [thanks!]. But my mailbox quotas are growing even after I go in and delete all the emails? I have a rule setup to forward the mail when received to another account, and delete it after 4 days using a bash script in a cron job. But even though the all the folders appear empty, ispconfig tells me there is about 100MB in there?
The mails are stored usually in /var/vmail/..... you can check there on the cli to find out real used sizes.
Many email programs do not remove emails when you delete them, they mark them just as removed and don't show them anymore instead of actually deleting them (to free up space), so no space is freed in that case. Most likely your mail program has an option to compact the mail folders or a similar function to really remove deleted mails.
