Mailbox size policy & Cyrus

Discussion in 'Developers' Forum' started by oliver.blaha, Apr 26, 2006.

  1. oliver.blaha

    oliver.blaha New Member


    I'm planning to implement a wrapper for Cyrus IMAP. I know, Cyrus is more complicated to install, but it has some advantages:

    - More scalable than Courier
    - Sieve mail filtering support
    - Easy quota checking, e.g. for webmail clients

    Integrating Cyrus support into ISPConfig won't be such a big deal, BUT:
    Cyrus doesn't use system quotas, as it's running with its own user and group. This way quotas have to be setup directly in cyrus. That itself isn't a problem, as this could be handled similar to mbox files, but with mbox, you can limit the size by putting the mailboxes on a quota partition - with cyrus there is no such possibility.

    This leads me to my question concerning a "mailbox size policy".

    There are some different options how it could be managed:

    1) The total size of all users' mailboxes (of one website) can't be more than the diskspace available to this web.
    1a) This total size is subtracted from the group quota. Disadvantage: You might have used already more than the new group quota
    1b) No subtraction takes place. Disadvantage: A website might use the double amount of space it was assigned.

    2) Do it like the new SQL quota. Resellers have a limit_mail_space or sth like that, websites have a web_mail_space and can assign their users mailbox space from this mail space pool.

    Advantage: Very flexible
    Disadv.: A lot of different quotas

    3) Resellers have just one total space limit, which includes sql, mail and web. Resellers then can setup webs:
    sum(web) + sum(sql) + sum(web_mail_space) <= limit_space

    Each website gets a web_mail_space quota, which is the maximum amount of space that can be dedicated to mailboxes:
    sum(mailboxes) <= web_mail_space

    Advantage: This gives (almost) maximum flexibility to resellers for their offers, but keeps handling resellers quite simple.
    Disadvantage: I would have to change the sql quota handling, as currently there is a seperate reseller limit for it (but I guess as it's just been in trunk one day right now this shouldn't break anything *g*)

    4) Just like 2 and 3 combined. One great total quota, but also seperate space quotas for web, mail, mysql. This way you can e.g. say "the reseller may use 1 GB of total space, but he may never assign more than 200 MB of database space"
    Advantage: You can do whatever you want to ;) Really maximum flexibility.
    Disadv.: Much harder to implement.

    What do you think?
    I like 3), especially because it could be extended to 4) in future without breaking anything.

    Hope you answer fast as I want to start as fast as possible... and I want to try getting it finished until friday.
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    Hi Oliver,

    I think 3) is a good compromise between easy handling of reselers and the ability to define fine grained quotas.

    Please make sure that this feature can be enabled / disabled by a setting in the system configuration (isp_server table) by either a checkbox or a select field where the POP3 / IMAP server type can be selected. As far as i know there is no easy way to anable mailquota with maildir + procmail, thats why we hide the field for maildir setups currently.
  3. oliver.blaha

    oliver.blaha New Member

    I thought about replacing the Maildir switch by a Mailsystem drop down list, if possible.
    But: I think it would be much easier and safer to have an option in, because in my opinion this should never be changed in a running system as you could screw up to much.

    I'd suggest to keep the Maildir option for compatibility, but introducing an option in - and if the latter is set, the option in the settings tab is ignored and hidden.

  4. till

    till Super Moderator Staff Member ISPConfig Developer

    I prefer if the settings remain in the isp_server table. It is easier to handle when the system is updated and almost all other options where stored there.

    I guess most options in the server settings will crash your system when changed after initial setup. Maybe we shall place a warning message there.

    Introducing 2 places for the same config setting where overrides the isp_server table may confuse poeple and will make support more difficult.
  5. oliver.blaha

    oliver.blaha New Member

    Just a little question is left:
    Shall I implement the quota behaviour for cyrus only? Or should mbox mailboxes be restricted by this total quota, too? (Or should I simply add another option to enable/disable the behaviour? If yes, should that be a config file option or also a setting in isp_server? I guess you'll answer the latter *g*)
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    I think the best way will be to implement the new behaviour for cyrus and mbox, so we can reuse the mail quota field in the isp_isp_user table too.
  7. falko

    falko Super Moderator Howtoforge Staff

    The main reason why we hide this field for Maildir is that Maildir is in the users' homedirs and therefore falls under the group quota of the web whereas mbox mailboxes may reside on another partition without quota enabled.

    I think we should also apply this to Maildir then, don't you think so?
  8. oliver.blaha

    oliver.blaha New Member

    I don't think this would make sense. I'm introducing a new value for limiting user_mailquota, and Maildir simply doesn't use that. Web quota of users shouldn't be touched by that, on the one hand because these are inside the boundaries of web space anyway, on the other hand because it's technically a WEB quota, that shouldn't be limited by some mail quota value.

    But: As soon as Maildir would get its own mailbox quota, just like mbox and cyrus, I agree.

    In short terms: I'd prefer to implement this behaviour for ALL mechanisms, but Maildir just doesn't use the value and thus isn't affected.

    Another question:
    As far as I understand the code, users are not really deleted in "user_delete", but only disabled, and get really purged in "web_user_clean". But the mbox file is already deleted in "user_delete"... Where should I delete cyrus mailboxes? Btw, it doesn't hurt creating mailboxes that already exist (e.g. when restoring).

    And last: I'm quite sure I'll finish cyrus work today or tomorrow, cyrus interface itself is complete, just some cosmetic changes to do. But I still have to implement some imit checks.

    Last question for this post: Cyrus IMAP is managed by an administrator user. This username is not hardcoded, but settable in /etc/imapd.conf. This means I need an option in ISPConfig to specify username and password of this user. Where should I add them? or isp_server? Maybe I should extend the installer to recognize if cyrus is installed, and if yes, ask during setup?
  9. till

    till Super Moderator Staff Member ISPConfig Developer

    If you can recreate the mailboxes easily when the user is restored, put the code in user_delete.

    Thats great. Please notify me when you checked in your changes, i would like to add some minor enhancements to the mail user form and wont break your large patch :)

    1 Vote for isp_server.

    That will be the optimum. Or you introduce a new variable in dist.txt which contains the default cyrus user of the linux distribution.
  10. oliver.blaha

    oliver.blaha New Member

    They can easily be recreated, but mails are lost when the mailbox is deleted. If I'm not completely wrong then currently mbox and Maildir behave different:

    mbox: Mails are erased when deleting the user and won't be restored if you restore the mailbox.
    Maildir: The mailbox is disabled when deleting, mail stays there until you empty the recycle bin.

    Is this intended? Or am I wrong? (I currently have no ISPConfig ready for testing with Maildir or mbox, so I have to ask *g*)
  11. oliver.blaha

    oliver.blaha New Member

    You can specifiy any user as administrator in /etc/imapd.conf, so this wouldn't really help. Anyway, you have to tell ISPConfig the password somewhere.

    Should I store it there also when asking for username and password during setup? If yes, should this be directly changeable through the server tab in ISPConfig? If also yes, should I hide theses two fields as long cyrus is not chosen? (Remember, if yes to the latter it would be necessary to reload the tab to show the fields)
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    The correct behaviour is to disable the user (if possible) when it is moved to the recycle bin and remove it completely, when the recycle bin is emptied.
  13. till

    till Super Moderator Staff Member ISPConfig Developer

    Ok, then the best way might be to ask during setup. Ask only in expert mode or ask always? Do you want to try to detect the used pop3 / IMAP damon or always ask which daemon is used?



    I think it is not nescessary to hide these fields.
  14. oliver.blaha

    oliver.blaha New Member

    Yes. Should be quite easy, if cyrdeliver (the cyrus delivery program) is installed I 'd assume Cyrus. Same should be true for some Courier binary.

    I'd choose standard mode, as it is a similar question like postfix and proftpd.

    I just see one problem: For automatic detection, I need to adapt the dists.txt file, but I don't have the possibility to examine the directory structure of all supported distributions. What's the best way to manage this?
  15. till

    till Super Moderator Staff Member ISPConfig Developer

    I guess the best way is to add the variables for all dists to the value of your current distribution and we will correct the values if nescessary when we test other linux dists.
  16. oliver.blaha

    oliver.blaha New Member

    I recently added cyrus support to subversion. I didn't touch the installer yet, cyrus has to be choosed in the isp_server tab, as I'm not really sure how the installer works.

    Setting up Cyrus itself is quite easy, but it's very hard to make procmail deliver correctly, as the procmailrc files are run with users' privileges, and these are not sufficient to deliver mails to cyrus (if they were, every local user could pipe unscanned mails to every mailbox).

    That means there is the choice between 2 possibilities:

    1) Don't use procmail. That would definitly speed up delivery for huge amounts of mails. Filtering could be done with a postfix content_filter or by a sieve script. I'd like doing it with sieve, but that would also mean that both, prochmail AND sieve scripts, would have to be maintained in future. Should not be too hard, but I wanted to mention that :D Also the MailScanner would have to be replaced by something else, as a procmail script won't work with sieve. But I already found a nice peace of software that might be a perfect substitution (and even cares of partial mails, which currently get through unscanned AFAIK) - I just forgot the name, but I know where I have to look for it *g*

    2) Run procmail scripts with delivery rights. Sounds simple, but it isn't that easy. Ok, running itself IS simple... but this way the users' .procmailrc files will also need to be run with these rights.
    This means that users could change their .procmailrc file and do something very odd if they wanted to.
    The rc files having root as owner and no write privileges for group and others doesn't help, as e.g. renaming the file and creating an own one is still possible.
    Only (known) solution: Store the users' .procmailrc files (and all other files called by it) in another, not accessible folder. As all the procmail files shouldn't be changed by users in any case, this seems to be a good solution. I'd make this the standard behavoiur if cyrus is selected.

    Comments please :)
  17. till

    till Super Moderator Staff Member ISPConfig Developer

    Hi Oliver,

    I think soultion 1) will be cleaner, but might cause incompatibility problems during upgrades.

    Solution 2) seems to be a bit more compatible with the current maildelivery implementation.

    If we use 1) an upgrade from current ISPCOnfig versions will be problematic as the user will have to reconfigure its postfix manually. And we will have to deliver new filters that are compatible with sendmail too.

    I think going with 2) will give us less upgrade problems.
    Last edited: May 1, 2006

Share This Page