migration AFTER migration has been done... does it affect new server?

EXACTLY your command till
and I see all the new emails under the correct user when I login. just cant delete anything!
cdb.

 

yes I vmailed everything. and I can send out an email from the roundcube client just fine! I just cant delete anything.
when I email out, mail is properly moved to the 'sent' folder.
/var/log/roundcube/roundcube.log shows nothing at all from any recent operation.

 

from darkest memory I remember creating a 99-ispconfig-custom-config.conf in /etc/dovecot/conf.d
I moved it to the new server (was not there) and problem has evaporated!
vsz_limit was the culprit.

 

for a bit of automation.
after saving the key on remote server with ssh-keygen and ssh-copy-id we can rsync all mail BUT the one I dont want with:

#!/bin/bash
ignore="/var/vmail/1stfamilyhomecareinc.com/"
for d in /var/vmail/*/ ; do
[ -L "${d%/}" ] && continue
if [ $d != $ignore ];
then
rsync -avz --delete -e ssh $d [email protected]:$d
fi
done

 

my script above seems to work nicely but I wonder about the cache folders do they need to be to rebuilt?
on a recent run I got:

laura/Maildir/dovecot.index.cache.tmp
WARNING: laura/Maildir/dovecot.index.cache failed verification -- update discarded (will try again).

this is referring to the target. its an error from rsync. any concern?

also since I created the certs for the rsync transfer - can I turn on 2FA for ssh? or will that break the rsync?

 

I have been using the rsync vmail folders to update mail on the new server (been too dam ill to complete migration) and I see one of the roundcube accounts says NO emails in the inbox (but lists it having 947312 emails - correctly!) other imap folders do have their emails. some cache file somewhere got messed up??
if I click on another folder, then return and click on inbox the contents of the inbox are shown properly!
but clicking on ANY of them gives a 'could not load message from server' error in red at the top!
seen this before?
looking in /var/log/roundcube/errors.log from yesterday I found this error at the end:

Code:

[06-May-2024 12:08:42 +0000]: <8t29vge1> DB Error: [1062] Duplicate entry '8t29vge11tdkm0di170577o042' for key 'PRIMARY' (SQL Query: INSERT INTO `session` (`sess_id`, `vars`, `ip`, `changed`) VALUES ('8t29vge11tdkm0di170577o042', 'dGVtcHxiOjE7bGFuZ3VhZ2V8czo1OiJlbl9VUyI7c2tpbl9jb25maWd8YTo3OntzOjE3OiJzdXBwb3J0ZWRfbGF5b3V0cyI7YToxOntpOjA7czoxMDoid2lkZXNjcmVlbiI7fXM6MjI6ImpxdWVyeV91aV9jb2xvcnNfdGhlbWUiO3M6OToiYm9vdHN0cmFwIjtzOjE4OiJlbWJlZF9jc3NfbG9jYXRpb24iO3M6MTc6Ii9zdHlsZXMvZW1iZWQuY3NzIjtzOjE5OiJlZGl0b3JfY3NzX2xvY2F0aW9uIjtzOjE3OiIvc3R5bGVzL2VtYmVkLmNzcyI7czoxNzoiZGFya19tb2RlX3N1cHBvcnQiO2I6MTtzOjI2OiJtZWRpYV9icm93c2VyX2Nzc19sb2NhdGlvbiI7czo0OiJub25lIjtzOjIxOiJhZGRpdGlvbmFsX2xvZ29fdHlwZXMiO2E6Mzp7aTowO3M6NDoiZGFyayI7aToxO3M6NToic21hbGwiO2k6MjtzOjEwOiJzbWFsbC1kYXJrIjt9fXJlcXVlc3RfdG9rZW58czozMjoiclN3aUFzTm5QWHRoTmRmWXJBc0hKOU9nWk1tYXdHb0oiOw==', '174.67.97.62', now())) in /usr/share/roundcube/program/lib/Roundcube/rcube_db.php on line 578 (GET /webmail/?_task=mail&_frame=1&_mbox=INBOX&_uid=15618&_part=2&_action=get&_extwin=1)

in /var/log/mail.log i'm seeing:

Code:

2024-05-07T08:12:47.959341-04:00 ns11 dovecot: imap([email protected])<3775622><ge+u5tsXGpDAqAIB>: Error: Transaction log /var/vmail/mandalaresearch.com/laura/Maildir/dovecot.list.index.log: duplicate transaction log sequence (7188)
2024-05-07T08:12:47.961611-04:00 ns11 dovecot: imap([email protected])<3775622><ge+u5tsXGpDAqAIB>: Error: Mailbox INBOX: Transaction log /var/vmail/mandalaresearch.com/laura/Maildir/dovecot.index.log: duplicate transaction log sequence (8884)
2024-05-07T08:12:47.961806-04:00 ns11 dovecot: imap([email protected])<3775622><ge+u5tsXGpDAqAIB>: Error: Mailbox INBOX: Transaction log /var/vmail/mandalaresearch.com/laura/Maildir/dovecot.index.log: duplicate transaction log sequence (8884)
2024-05-07T08:12:47.961888-04:00 ns11 dovecot: imap([email protected])<3775622><ge+u5tsXGpDAqAIB>: Error: Mailbox INBOX: Transaction log /var/vmail/mandalaresearch.com/laura/Maildir/dovecot.index.log: duplicate transaction log sequence (8884)
2024-05-07T08:12:47.961965-04:00 ns11 dovecot: imap([email protected])<3775622><ge+u5tsXGpDAqAIB>: Error: Transaction log /var/vmail/mandalaresearch.com/laura/Maildir/dovecot.list.index.log: duplicate transaction log sequence (7188)

any ideas??

 

I turned on the imap.log file, but dont see anything bogus:
and if you click on emails in any of the other imap folders they display fine! the could not load message from server seems only to affect the inbox

Code:

-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * 0 RECENT
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [UIDVALIDITY 1444654856] UIDs valid
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [UIDNEXT 4] Predicted next UID
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [HIGHESTMODSEQ 2] Highest
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: A0067 OK [READ-WRITE] Select completed (0.001 + 0.000 secs).
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] C: A0068 UID SEARCH 3
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * SEARCH 3
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: A0068 OK Search completed (0.001 + 0.000 secs).
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] C: A0069 STATUS "Follow up" (MESSAGES UNSEEN)
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * STATUS "Follow up" (MESSAGES 2 UNSEEN 0)
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: A0069 OK Status completed (0.001 + 0.000 secs).
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] C: A0070 SELECT "Follow up"
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [CLOSED] Previous mailbox closed.
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * 2 EXISTS
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * 0 RECENT
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [UIDVALIDITY 1444654880] UIDs valid
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [UIDNEXT 3] Predicted next UID
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [HIGHESTMODSEQ 4] Highest
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: A0070 OK [READ-WRITE] Select completed (0.001 + 0.000 secs).
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] C: A0071 UID SEARCH 2
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * SEARCH 2
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: A0071 OK Search completed (0.001 + 0.000 secs).
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] C: A0072 STATUS "Non work related" (MESSAGES UNSEEN)
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * STATUS "Non work related" (MESSAGES 0 UNSEEN 0)
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: A0072 OK Status completed (0.001 + 0.000 secs).
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] C: A0073 STATUS Notes (MESSAGES UNSEEN)
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * STATUS Notes (MESSAGES 3 UNSEEN 0)
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: A0073 OK Status completed (0.001 + 0.000 secs).
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] C: A0074 SELECT Notes
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [CLOSED] Previous mailbox closed.
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * 3 EXISTS
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * 0 RECENT
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [UIDVALIDITY 1444654860] UIDs valid
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [UIDNEXT 4] Predicted next UID
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [HIGHESTMODSEQ 1] Highest
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: A0074 OK [READ-WRITE] Select completed (0.001 + 0.000 secs).
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] C: A0075 UID SEARCH 3
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * SEARCH 3
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: A0075 OK Search completed (0.001 + 0.000 secs).
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] C: A0076 STATUS Spam (MESSAGES UNSEEN)
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * STATUS Spam (MESSAGES 5 UNSEEN 1)
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: A0076 OK Status completed (0.001 + 0.000 secs).
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] C: A0077 SELECT Spam
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [CLOSED] Previous mailbox closed.
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * 5 EXISTS
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * 0 RECENT
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [UNSEEN 3] First unseen.
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [UIDVALIDITY 1444654844] UIDs valid
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [UIDNEXT 11] Predicted next UID
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [HIGHESTMODSEQ 6] Highest
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: A0077 OK [READ-WRITE] Select completed (0.001 + 0.000 secs).
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] C: A0078 UID SEARCH 5
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * SEARCH 5
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: A0078 OK Search completed (0.001 + 0.000 secs).
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] C: A0079 SELECT INBOX
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [CLOSED] Previous mailbox closed.
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-3 unknown-0 unknown-1 unknown-2 $Forwarded $MDNSent $Junk Forwarded $MailFlagBit0 Junk $ReadReceipt)
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-3 unknown-0 unknown-1 unknown-2 $Forwarded $MDNSent $Junk Forwarded $MailFlagBit0 Junk $ReadReceipt \*)] Flags permitted.
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * 947321 EXISTS
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * 0 RECENT
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [UNSEEN 455987] First unseen.
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [UIDVALIDITY 1444654842] UIDs valid
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [UIDNEXT 1092252] Predicted next UID
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * OK [HIGHESTMODSEQ 784111] Highest
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: A0079 OK [READ-WRITE] Select completed (0.619 + 0.000 + 0.618 secs).
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] C: A0080 LOGOUT
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: * BYE Logging out
[07-May-2024 12:35:28 +0000]: <2s76sce7> [4A7A] S: A0080 OK Logout completed (0.050 + 0.000 + 0.049 secs).

some memory limit error?

 

I note someone else with similar issues saying problem was that an rsync was running while someone was in roundcube. that is certainly possible I've been rsyncing the mail folders every 30 minutes. i've stopped that now
they suggested using dsync instead of rsync?

 

some information from tail /var/log/mail.log:

Code:

2024-05-09T11:19:44.239678-04:00 ns11 dovecot: imap([email protected])<103509><o3By8gYYaIQAAAAAAAAAAAAAAAAAAAAB>: Panic: file mail-index-util.c: line 10 (mail_index_uint32_to_offset): assertion failed: (offset < 0x40000000)
2024-05-09T11:19:44.244824-04:00 ns11 dovecot: imap([email protected])<103509><o3By8gYYaIQAAAAAAAAAAAAAAAAAAAAB>: Error: Raw backtrace: #0 test_subprocess_fork[0x7f997ee1ea20] -> #1 backtrace_append[0x7f997ee1ec90] -> #2 backtrace_get[0x7f997ee1ee20] -> #3 execvp_const[0x7f997ee2bf90] -> #4 i_syslog_fatal_handler[0x7f997ee2c900] -> #5 i_panic[0x7f997ed820a4] -> #6 mail_index_sync_set_corrupted[0x7f997ef4b8e5] -> #7 mail_cache_get_first_new_seq[0x7f997efe52d0] -> #8 mail_cache_purge[0x7f997efe6590] -> #9 mail_index_sync_commit[0x7f997eff1e30] -> #10 maildir_sync_index_begin[0x7f997ef8a3a0] -> #11 maildir_list_index_update_sync[0x7f997ef8c1f0] -> #12 maildir_list_index_update_sync[0x7f997ef8c6a0] -> #13 maildir_storage_sync_force[0x7f997ef8c7a0] -> #14 maildir_file_do[0x7f997ef92a70] -> #15 maildir_sync_is_synced[0x7f997ef87280] -> #16 mail_get_stream_because[0x7f997ef55dd0] -> #17 index_mail_prefetch[0x7f997efcc010] -> #18 mail_prefetch[0x7f997ef512a0] -> #19 index_search_get_mail[0x7f997efd42a0] -> #20 index_storage_search_next_nonblock[0x7f997efd4800] -> #21 mailbox_search_next_nonblock[0x7f997ef627d0] -> #22 mailbox_search_next[0x7f997ef62820] -> #23 imap_fetch_begin[0x56448a3e1170] -> #24 imap_fetch_more[0x56448a3e1550] -> #25 cmd_fetch[0x56448a3d5c00] -> #26 command_exec[0x56448a3de880] -> #27 cmd_x_cancel[0x56448a3e4510] -> #28 cmd_x_cancel[0x56448a3e4510] -> #29 cmd_x_cancel[0x56448a3e4510] -> #30 client_handle_input[0x56448a3e4880] -> #31 client_input[0x56448a3e4d80] -> #32 io_loop_call_io[0x7f997ee43c70] -> #33 io_loop_handler_run_internal[0x7f997ee45970] -> #34 io_loop_handler_run[0x7f997ee45b00] -> #35 io_loop_run[0x7f997ee45cd0] -> #36 master_service_run[0x7f997edb6180] -> #37 main[0x56448a3d0a00] -> #38 __libc_init_first[0x7f997eb6c1d0] -> #39 __libc_start_main[0x7f997eb6c280] -> #40 _start[0x56448a3d1000]
2024-05-09T11:19:44.346794-04:00 ns11 dovecot: imap([email protected])<103509><o3By8gYYaIQAAAAAAAAAAAAAAAAAAAAB>: Fatal: master: service(imap): child 103509 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps)

ouch! I restarted dovecot, but what do we do to fix this???

 

ENLIGHTENMENT!
the dovecot.index.cache file in the /var/.../Maildir directory was over 2gb in size.
deleting it seems the problem has gone away seems its a known problem in roundcube!
from a comment on a thread:
Known issue when folder cache is too big. Try rm -rf dovecot.index.cache
for the folder.

 

is there any known fix for the roundcube problem where dovecot.index.cache causes it to fail to load messages? deleting the file does allow the messages to be reloaded, but soon thereafter we get the same error again!
any known solution? just deleting the file all the time seems a poor fix!

 

its only an issue with a HUGE inbox. in this case customer has 400k UNREAD messages LOL
it is also giving a "server failed to send fetch command". research indicates that setting the list option away from and then back to NONE under sorting column may ameliorate this error.