Emails Just Stopped Working

The email system suddenly stopped working today.

Postfix and Dovecot are running. What I'm seeing with "tail -f /var/log/mail.log" and trying to connect to one of the email accounts with my phone is lots of disconnections:

"dovecot: disconnection before auth was ready"
"smtpd: lost connection after CONNECT from..."
"imap: connection closed"

So it appears that all connections to postfix / dovecot are being prematurely cut off almost immediately.

When I try to send an email to or from the email account, it's not delivered but it also doesn't bounce either. It just vanishes into the ether.

Is this SpamAssassin or Fail2Ban getting far too eager - forcing a disconnect - and actually banning any emails from working at all?

 

Although, that said, amavis / SpamAssasin also writes to "mail.log" too, yes? But I'm not seeing anything from them. And a Fail2Ban on an IP address would also knock out the websites, but those are up.

How would I root out what's causing this in order to fix it?

I can see that the connections to postfix / dovecot are all being prematurely closed (but not to apache on the same nodes, as the websites are up). But what could be causing that to happen?

 

Wait, thinking about it, the periodic connections and disconnections are likely LVS doing its "healthcheck" to see that the node is still up and running. I didn't configure that to login, but only to test that it could get a connection.

 

Yes, I worked that out by my own investigations.

And now I've got the healthchecks actually logging into a "[email protected]" account for email stuff. That's both a better healthcheck - as it's not just checking for a connection, but that the service actually works - and it allows me to spot the difference between the healthchecks and actual emails by email address.

Though it would be lovely if I could somehow get the healthchecks logged to a different file, so it doesn't get in the way of seeing actual errors in the logs.

So if it's not that, then why are the emails no longer working?

I've been pulled in on my day off to sort this out!

 

does amavis write anything to your log if you restart that service? does postfix give any hints about issues with the config if you restart it?
usually thing's don't just stop to work, there has to be a reason usually - or maybe the services were cut off due to resource outage? Is it on a virtual machine?

if you can connect to the server, you're not banned

 

The problem was those spammy healthchecks from the load balancer made reading the mail log hard to follow.

But once I slowed the checks down, I tried sending an email (from one of the email accounts to itself) and could see the message from "amavis" declaring that "sender address triggered FILTER". SpamAssassin was thinking everything was spam.

So I went into the config file and changed the policy from "DISCARD" to "PASS". It's all coming through marked as junk mail, but at least it's coming through and that'll have to do until Monday, as I don't work weekends.

(The irony is that, the night before, I'd ordered a book on Postfix and Dovecot from Amazon to do some reading up for next week, when I intended to turn my attentions to properly sorting out the email system. As the boss noted, it is Friday the 13th today.)

 

I'm sure that I changed the policy from "DISCARD" to "PASS" before, though.

(As I think that marking emails as spam and passing them on is the safest default, in case SpamAssassin wrongly diagnoses spam. Let the user make the choice of what to delete.)

Would an upgrade of ISPConfig - as I did do that to ensure that it was on the latest version before copying over my interface modifications, as I'd modified the latest version and wanted them to be the same before copying - have potentially reset the amavis policy settings like that?

 

Okay, then it was something else I did. Or perhaps I only altered it on one node and not all of them. Or maybe I did it on my laptop but not on the servers.

But I do distinctly remember doing this before - having to look up where the amavis policies were stored and changing them - yet they were "undone" when I looked. Well, whatever it was, that's what was causing the issues and, next week, I intend to properly go over the email system and will systematically check everything in order (up until now, I've just been leaving things at the defaults, while I dealt with everything else, as we've not been actively selling any email packages to anyone, mostly offer to resell third-party email when someone insists they need it and there's only a few we deal with ourselves - sorting it out so it's an actual product worthy of sale was the next big thing to deal with on my list, but irony would insist for it to go horribly wrong the week before I was scheduled to concentrate on it).


(p.s. Getting the DNS ready was really preparation for all the MX, SPF and DMARC records and all that sort of thing and having them under our control for ease and speed of making DNS changes. I was sorting that, so that it'd be ready for sorting out the emails later.)