Dear ISPConfig experts/developers, please comment on the below solution and help me to undertand are there any missing or troublesome parts, so I can put it into production: We have to migrate mails from very old IMAP/POP server to ISPConfig and one of major problems was that mailboxes on the old server have mbox format, while in ISPConfig Maildir is used. I have played much with some Mailbox-to-Maildir converting methods like Mb2MD, Dsync, ImapCopy, ... but none of them gave the expected result, since we wanted to automate migration of hundreds of large sized mailboxes and each tool failed for some mailbox/message losing some mails & not working perfectly. I have thought about one possibility, that I want to share and ask to comment on, if its good. The idea is to make ISPConfig (including dovecot and other parts) to work with mbox format instead of maildir. I have found the following workaround. Since in ISPConfig setup Dovecot is getting the mailbox format from 'mail_user' MySQL table, I have changed the following line in /etc/dovecot/dovecot-sql.conf: user_query = SELECT email as user, maildir as home, CONCAT('maildir:', maildir, '/Maildir') as mail, uid, gid, CONCAT('*:storage=', quota, 'B') AS quota_rule, CONCAT(maildir, '/.sieve') as sieve FROM mail_user WHERE (login = '%u' OR email = '%u') AND `disable%Ls` = 'n' AND server_id = '1' to the following: user_query = SELECT email as user, maildir as home, CONCAT('mbox:', maildir, '/Mbox') as mail, uid, gid, CONCAT('*:storage=', quota, 'B') AS quota_rule, CONCAT(maildir, '/.sieve') as sieve FROM mail_user WHERE (login = '%u' OR email = '%u') AND `disable%Ls` = 'n' AND server_id = '1' Next I wrote small shell script to move each user's mailboxes (in mbox format) to their 'Mbox' directory for a particular domain in /var/vmail. For each user it would be: /var/vmail/$DOMAIN/$USER/Mbox In our old server all mailboxes were kept in user's home directory in 'mail' subdirectory, where '.subscriptions' file was present too, so moving all that stuff to '/var/vmail/$DOMAIN/$USER/Mbox' also keeps user's IMAP folder subscriptions as well. Also user's INBOX should be moved in the same directory and should be named: /var/vmail/$DOMAIN/$NEWUSERN/Mbox/inbox Additionally I do: chown -R vmail:vmail /var/vmail/$DOMAIN/$USER/Mbox After restarting dovecot it seems to start working. Will this work or I miss anything? Thanks in advance for any useful comment
I guess this should work. Just be careful with updates, you might want to take the dovecot-sql.conf master template from install/tpl folder of the ospconfig.tar.gz file, modify it and store under /usr/local/ispconfig/server/conf-custom/install/ so that the updater can find it there.
Thanks a lot for quick reply. I have created such one in /usr/local/ispconfig/server/conf-custom/install/ Just the filename was 'debian_dovecot-sql.conf.master' (platform name added in the begining), I hope that's the one you told about, since I use Debian 8. What I want next is to build a Cluster as described at: https://www.howtoforge.com/tutorial...ase-cluster-on-debian-8.4-with-ispconfig-3/5/ but on the second node leave the storage as 'maildir'. Will that do the trick and give me all the mailboxes converted? Also could you please suggest about 'mdbox', do I need to think about that and where to read more about it so I can understand if I need to move directly from 'mailbox' to 'mdbox' instead of 'maildir' Thanks a lot for assistance in advance
I don't think that dovecot can work in a mirror with different storage types, but you might have to ask the dovecot developers if this is possible. ISPconfig does not use mbox because large mboxes can get corrupted, maildir is more fault tolerant and used by most systems. mdbox is a mixture of mbox and maildir, the drawback is that you have to be careful to not lose any of the mdbox file as you can not rebuild the indexes, see dovecot docs for the different formats and their pro's and cons. Personally, I use maildir and don't have any issues with it even with very large mail accounts.
Thanks a lot for useful comments. I would prefer Maildir too, but don't know how to find good working solution to convert my large mailboxes /about 500Gb/, and do that without stopping the mail service to work and serve users with the coming new mails too. If you have any working idea I would be thankful. My final goal is to migrate mailserver to ISPConfig.
I've used imapsync with 10G-20G mailboxes (never 500G), and it has done nicely. You can have it cache messages on your local machine (ie. wherever you run imapsync from) and rerun it over and over to update changes from one system to the other. You may need to play with imapsync options to get things working on both imap servers and the dates copied correctly.
Thanks Jesse, Could you please post a link to the 'imapsync' variant you used (there are several different scripts with similiar name) and also examples of parameters/options you used to do initial run and re-runs. This would help me much, because I have done much similar tests but didn't get desired result. Also please explain how do you deal with passwords? Because on my old IMAP server there is no possibility to use a master password, so if I even reset temporarily passwords for users to migrate their mailboxes, theu would not be able to use it anymore. So, the problem is leaving user's access working and migrating a huge data transparently. Any ideas?
Have you checked their website? Most projects have a mailinglist, forum or other ways to contact the developers.
Maybe you can try http://batleth.sapienti-sat.org/projects/mb2md/ to switch from mdbox to maildir. But you can run dovecot with mdbox without any problems. It's a little bit different to maildir (espacilly for backup / restore) but are created much faster. iirc ispconfig 3.1 can handle mdbox.
Thanks Florian, I have tried mb2md, but it take much time to convert my huge amount, so I think to just move mails from old mailserver leaving it with mbox format. Hope first to migrate to ISPConfig with mbox and then maybe find some solution to change mailbox format too. If you have mdbox experience, please post some links where to read more about that format. Another thing I would like to ask to Till. The following post contains small extension to import mail accounts from csv: https://www.howtoforge.com/community/threads/bulk-import-of-mail-accounts.63768/ It works fine with ISPConfig3, but doesn't work with 3.1. Can you help me in getting it work in 3.1 too. It installs OK, but does nothing. Would be very thankful to Till in case of help.
It's been a few years (2013), and I don't remember exactly where I got obtained it, but: Code: $ imapsync --version 1.536 You have to figure out the parameters that work for your imap servers, so read the help/man page, search google for it, and/or use trial and error, but what I used (going from dbmail 1.2 to dovecot 1.1 or 1.2 I think) was something like: Code: $ cat imapsync-example.sh #!/bin/sh d=domain.com h1=11.22.33.44 # src imap server h2=22.33.44.55 # dst imap server u=username1 p=passwd1 imapsync --noreleasecheck --tmpdir /var/tmp --useuid --nofoldersizes --pidfile /var/tmp/imapsync-${u}-${d}.pid \ --pidfilelocking --syncinternaldates --tls1 --tls2 \ --host1 $h1 --authmech1 LOGIN --user1 ${u}@${d} --password1 ${p} \ --host2 $h2 --authmech2 LOGIN --user2 ${u}@${d} --password2 ${p} u=username2 p=passwd2 imapsync --noreleasecheck --tmpdir /var/tmp --useuid --nofoldersizes --pidfile /var/tmp/imapsync-${u}-${d}.pid \ --pidfilelocking --syncinternaldates --tls1 --tls2 \ --host1 $h1 --authmech1 LOGIN --user1 ${u}@${d} --password1 ${p} \ --host2 $h2 --authmech2 LOGIN --user2 ${u}@${d} --password2 ${p} I would use something like that for a few accounts where typing username/password wasn't difficult. On larger runs (hundreds or thousands of users) I'd export username/passwords to a text file an read from there. It looks like I would run 3 imapsync's simultaneously, but how many you can do at once will depend on your resources (primarily disk speeds). Also when you are going to re-sync a mailbox you've sync'd previously, you probably want to add --delete2 and maybe --delete2folders, just don't use those after you switch smtp delivery to the new server if you do any more resync's after that (or you'll delete the new mail that came in to the new server). You can test things with --dry without doing any harm, and of course use a small test account to get things tested/working. You have to be able to login to both imap servers, so if you can't get the user passwords on the old side, you'll need to do something in the imap server itself to let you login. Eg. hack it to always accept a specific (very long) password for every account, or always allow auth for your ip addr. I think on the larger migrations we captured the passwords of all active accounts when they logged in for a couple months in order to be able to set them on the new server, and then simply didn't worry about the inactive accounts (they'll call in for support if/when they try to check mail again, but by selection they haven't been using their mail account in months, so aren't likely to start).
