I pulled my hair a few hours with this, so hope this helps others. User had several mailboxes, and used Thunderbird as e-mail client. I removed three of the mailboxes from server. Those e-mail accounts remain on Thunderbird, which tries to use them still. This leads to failed logins and fail2ban bans the IP. This had the hilarious consequence that all users mailboxes stopped working since Thunderbird could no longer connect to the mail server. Same thing on users laptop, since it was using the same public IP- number. Thunderbird did inform it could not access e-mail server, but I did not understand why. So: when removing mailboxes, first remove those e-mail accounts from users e-mail program. On all computers used for reading e-mails.