Dovecot sync problem in multiserver configuration after upgrading to Debian 12

Which is really frustrating. I'm still wating for some new info from the project. Hopefully there is at least some consideration on how to transition from systems with active replication setups to something else.

 

Just found a presentation from Heinlein which has some good information on the topic. Be aware this is all in german.

 

There is microceph which is designed for such use cases but we'll have to have a look into it as i'm not very familiar with it, just stumbled upon it recently. https://canonical-microceph.readthedocs-hosted.com/en/reef-stable/

 

For the question reported in the thread above about the possible removal of replication features in dovecot, I contacted Open-Xchange and they replied that they don't know about it.

I also report here the answer they gave me:
"I think there was a misunderstanding. Sync will only be removed in the Pro version starting from version 3.0 and replaced by the mandatory use of object storage (S3, Scality, etc.). The new Dovecot Pro version, although announced by March 2023 , has not yet been released and we have not been given a firm release date.
As far as I know, nothing changes in the Dovecot Community Edition. Then you should be able to continue working with the current Dovecot version as usual."

So if it were true there shouldn't be any problems in the future.

 

That is strange. There is even a table of features that get removed in 2.4/3.0 in the upgrade document on their webseite.
https://doc.dovecot.org/3.0/installation_guide/upgrading/from-2.3-to-3.0/

And other sources also mention this:

Can we somehow double check? @gscaglia Can you ask your contact again and ask them why it is listed in the offical documentation then?

 

ChatBOT GPT-4o says this to my detailed question on the matter:
"The feature mentioned as "replicator" in the Dovecot 3.0 documentation refers specifically to the Dovecot replicator service, which manages automated replication of users' mail. This functionality is indeed being removed in version 3.0, with the recommendation to use NFS or another shared filesystem instead. This change impacts both the community and pro editions of Dovecot.
The confusion likely arises from the close relationship between "replicator" and the "dsync" command used for replication. While the dsync command itself is not being removed, the -D parameter for dsync is being deprecated, and the symlink to doveadm sync is being removed, consolidating the functionality under doveadm sync."

Which essentially agrees with the response received and also with what I understand (but I'm not an expert) reading that page.

In any case, I wrote to my contact again in order to have a more accurate human confirmation and I will let you know as soon as he replies

 

I received a very clear response from OX (Open-Xchange) and I believe that the misunderstanding can be considered closed.

My interlocutor, an OX reseller, sent me the exchange he had directly with OX, asking me to summarize without directly posting what I received because it was an internal exchange.

The first answer, already very clear, I followed up with a second request for further information to which I attached our dovecot.conf configuration file and this how-to https://www.howtoforge.com/tutorial/ispconfig-multiserver-setup-debian-ubuntu/3/ to be absolutely sure that our configurations exclusively use doveadm-sync technology i.e. dsync because the whole issue revolves around this.

The good news is that certainly nothing has changed for dsync and nothing will change in the future, this was reiterated to me unequivocally.

The misunderstanding arises from the fact that OX does not consider dsync a real replication functionality but simply a way of copying email messages between two servers which, when used bidirectionally (server 1 copies to server 2 and server 2 copies to the server 1) emulates a replication functionality.

The real replication technologies according to them are Dovecot Director (which will be discontinued in CE and replaced in PRO by the Palomar architecture) and Replication or Replicator (which is discontinued in CE and has never been supported in PRO): both have nothing to what to do with dsync, and are the subject of the discontinuation communication from the CE version that generated the misunderstanding.

So dsync will be supported in the future by the CE version exactly as it has been until now.

Without prejudice to all this, I make a personal observation that is inspired by what happened in Debian 12 with the dovecot 2.3.19.1 version (synchronization problems with the opening of new mailboxes) and by the insistence with which OX, in all the replies received by me, limits dsync to the simple mail migration functionality as if the bidirectional copy were almost a stretch (which is instead fully foreseen on this official page https://doc.dovecot.org/3.0/man/doveadm-sync.1/).

There could be a conflict of interest between the bidirectional copy functionality offered by the CE with dsync (which as we know emulates replication perfectly) and the commercial PRO version which contains the actual replication technology and this conflict of interest could be the basis of the unclear communication that generated the misunderstanding.

So we can safely continue to use dsync, but let's keep our eyes open to understand whether it was exclusively a simple problem in communication or whether it is a sign of possible future changes (which today, however, as far as dsync is concerned, are absolutely excluded by OX) .

I hope I have been useful, the decisions regarding the consequences of all this on ISPConfig are up to you.

 

Sebas11 in the post above seems to have experimented with overcoming the problem of the lack of synchronization of new mailboxes present in versions 2.3.18 to 2.3.19.1 of dovecot by installing the dovecot version present in Debian testing.

I will preview this solution in August.

 

Thank you for the response and follow up.

@till @Th0m If this is how things turn out to be and we only use https://doc.dovecot.org/configuration_manual/replication/ for master / master replication we should be fine right? Or do you see any issues?

 

Well i was confused aswell. But @gscaglia mentioned the configuration and link to the howto to OX so it seems like that this is only using dsync and not whateven they called replication then?

 

OK so whatever the contact at OX is talking seems to be wrong. Everything in the documentation aswell as all topics in the mailling list mention the removal of replicator.

https://dovecot.org/mailman3/archiv....org/thread/2CPFZ5OXVA2QYHQBWH7P6QM4J4D7FEYE/
https://dovecot.org/mailman3/hyperk....org/thread/3QSDHEKVTQ7YJIU7OB5FPRP44QD4TTBQ/
https://doc.dovecot.org/3.0/installation_guide/upgrading/from-2.3-to-3.0/

Commit with Removal:
https://github.com/dovecot/core/com...3c749d14d42b25a9d60f0f149bface862f5ff348412c8

 

I even posted the configuration code for dovecot.conf to my interlocutor in OX and he seems to have taken it into account in the last reply.

But continuing my research I believe it is unlikely that dsync, in this precise passage in dovecot.conf, does not rely on what they call "replicator" and which will certainly be deprecated:

Code:

service replicator {
        process_min_avail = 1
        unix_listener replicator-doveadm {
                mode = 0666
        }

However, on the page https://doc.dovecot.org/3.0/man/doveadm-sync.1/ which should refer to version 3.0 of dovecot including the Community Edition, there is never any mention of "replicator" or even of replication but always and only synchronization and it seems that this happens automatically:
- doveadm-sync - Dovecot's two-way mailbox synchronization feature
- doveadm sync performs two-way synchronization. It merges all changes without losing anything. Both the mailboxes will end up looking identical after the synchronization is finished.

From my point of view, which is not very technical, it is necessary to understand how dsync works: asking my contact in OX again I think is useless, I leave the conclusions to you.

 

Dsync is not removed, which is true. The replicator plugin makes use of dysnc but automates the whole process of syncronizing two dovecot servers.

The commit is clearly showing the removal of the whole replicator plugin:
https://github.com/dovecot/core/com...3c749d14d42b25a9d60f0f149bface862f5ff348412c8

Sure you could run dsync manually with a cronjob but that is, if even, only working in very very small setups with a few mailboxes.